Image processing apparatus

ABSTRACT

An object of the present invention is to make it possible to adjust the amount of motion blur contained in a blurred image. A unit-of-processing determining portion  901  determines the unit of processing which is formed of pixel data located on a straight line. An equation generator  903  generates simultaneous equations consisting of a plurality of relational expressions based on the unit of processing. A calculator  904  generates foreground object components in which the amount of motion blur is adjusted by solving the simultaneous equations. The present invention can be applied to an image processing apparatus in which a difference between a signal detected by a sensor and the real world is taken into consideration.

TECHNICAL FIELD

[0001] The present invention relates to image processing apparatuses,and more particularly, to an image processing apparatus in which adifference between a signal detected by a sensor and the real world istaken into consideration.

BACKGROUND ART

[0002] A technique for detecting incidents occurring in the real worldby a sensor and for processing sampled data output from the image sensoris widely used.

[0003] For example, motion blur occurs in an image obtained by capturingan object moving in front of a predetermined stationary background witha video camera if the moving speed is relatively high.

[0004] Hitherto, in order to prevent such motion blur, for example, thespeed of an electronic shutter is increased so as to decrease theexposure time.

[0005] However, in the method in which the shutter speed is increased,it is necessary to adjust the shutter speed of a video camera beforecapturing an image. Accordingly, there is a problem in which blurredimages cannot be corrected to obtain sharp images.

DISCLOSURE OF INVENTION

[0006] The present invention has been made in view of theabove-described background. Accordingly, it is an object of the presentinvention to make it possible to adjust the amount of motion blurcontained in a detection signal of, for example, a blurred image.

[0007] An image processing apparatus of the present invention includes:unit-of-processing determining means for determining a unit ofprocessing, based on image data and area information indicating aforeground area consisting of foreground object components which form aforeground object in the image data, a background area consisting ofbackground object components which form a background object in the imagedata, and a mixed area in which the foreground object components and thebackground object components in the image data are mixed, the mixed areaincluding a covered background area formed at a leading end in adirection in which the foreground object is moving and an uncoveredbackground area formed at a trailing end in the direction in which theforeground object is moving, the unit of processing being formed ofpixel data located on at least a straight line that coincides with themoving direction of the foreground object and ranging from an outer endof the covered background area to an outer end of the uncoveredbackground area based on the foreground area; simultaneous-equationgenerating means for generating simultaneous equations consisting of aplurality of relational expressions by setting pixel values of pixelswithin the unit of processing determined based on the unit of processingand by setting a divided known value obtained by dividing the foregroundobject components in the mixed area by a set number of divided portions;and calculation means for calculating the foreground object componentsin which the amount of motion blur is adjusted by solving thesimultaneous equations.

[0008] By utilizing a characteristic in which the foreground objectcomponents contained in the first pixel from an end of the mixed areaare subtracted from the foreground object components contained in thesecond pixel, which is located adjacent to the first pixel, from the endof the mixed area on the straight line so that the single foregroundobject component corresponding to the second pixel is calculated, thecalculation means may sequentially solve the simultaneous equations fromthe relational expressions corresponding to the pixel located at theend, thereby calculating the foreground object components in which theamount of motion blur is adjusted.

[0009] The simultaneous-equation generating means may generate thesimultaneous equations based on the number of divided portions inaccordance with the amount of movement of the foreground object.

[0010] An image processing method of the present invention includes: aunit-of-processing determining step of determining a unit of processing,based on image data and area information indicating a foreground areaconsisting of foreground object components which form a foregroundobject in the image data, a background area consisting of backgroundobject components which form a background object in the image data, anda mixed area in which the foreground object components and thebackground object components in the image data are mixed, the mixed areaincluding a covered background area formed at a leading end in adirection in which the foreground object is moving and an uncoveredbackground area formed at a trailing end in the direction in which theforeground object is moving, the unit of processing being formed ofpixel data located on at least a straight line that coincides with themoving direction of the foreground object and ranging from an outer endof the covered background area to an outer end of the uncoveredbackground area based on the foreground area; a simultaneous-equationgenerating step of generating simultaneous equations consisting of aplurality of relational expressions by setting pixel values of pixelswithin the unit of processing determined based on the unit of processingand by setting a divided known value obtained by dividing the foregroundobject components in the mixed area by a set number of divided portions;and a calculation step of calculating the foreground object componentsin which the amount of motion blur is adjusted by solving thesimultaneous equations.

[0011] By utilizing a characteristic in which the foreground objectcomponents contained in the first pixel from an end of the mixed areaare subtracted from the foreground object components contained in thesecond pixel, which is located adjacent to the first pixel, from the endof the mixed area on the straight line so that the single foregroundobject component corresponding to the second pixel is calculated, thecalculation step may solve the simultaneous equations from therelational expressions corresponding to the pixel located at the end,thereby calculating the foreground object components in which the amountof motion blur is adjusted.

[0012] The simultaneous-equation generating step may generate thesimultaneous equations based on the number of divided portions inaccordance with the amount of movement of the foreground object.

[0013] A program of a recording medium of the present inventionincludes: a unit-of-processing determining step of determining a unit ofprocessing, based on image data and area information indicating aforeground area consisting of foreground object components which form aforeground object in the image data, a background area consisting ofbackground object components which form a background object in the imagedata, and a mixed area in which the foreground object components and thebackground object components in the image data are mixed, the mixed areaincluding a covered background area formed at a leading end in adirection in which the foreground object is moving and an uncoveredbackground area formed at a trailing end in the direction in which theforeground object is moving, the unit of processing being formed ofpixel data located on at least a straight line that coincides with themoving direction of the foreground object and ranging from an outer endof the covered background area to an outer end of the uncoveredbackground area based on the foreground area; a simultaneous-equationgenerating step of generating simultaneous equations consisting of aplurality of relational expressions by setting pixel values of pixelswithin the unit of processing determined based on the unit of processingand by setting a divided known value obtained by dividing the foregroundobject components in the mixed area by a set number of divided portions;and a calculation step of calculating the foreground object componentsin which the amount of motion blur is adjusted by solving thesimultaneous equations.

[0014] By utilizing a characteristic in which the foreground objectcomponents contained in the first pixel from an end of the mixed areaare subtracted from the foreground object components contained in thesecond pixel, which is located adjacent to the first pixel, from the endof the mixed area on the straight line so that the single foregroundobject component corresponding to the second pixel is calculated, thecalculation step may solve the simultaneous equations from therelational expressions corresponding to the pixel located at the end,thereby calculating the foreground object components in which the amountof motion blur is adjusted.

[0015] The simultaneous-equation generating step may generate thesimultaneous equations based on the number of divided portions inaccordance with the amount of movement of the foreground object.

[0016] An imaging apparatus of the present invention includes: imagingmeans for outputting a subject image as image data which is formed of apredetermined number of pixel data, the subject image being captured byan imaging device including a predetermined number of pixels, each pixelhaving a time integrating function; unit-of-processing determining meansfor determining a unit of processing, based on the image data and areainformation indicating a foreground area consisting of foreground objectcomponents which form a foreground object in the image data, abackground area consisting of background object components which form abackground object in the image data, and a mixed area in which theforeground object components and the background object components in theimage data are mixed, the mixed area including a covered background areaformed at a leading end in a direction in which the foreground object ismoving and an uncovered background area formed at a trailing end in thedirection in which the foreground object is moving, the unit ofprocessing being formed of pixel data located at least a straight linethat coincides with the moving direction of the foreground object andranging from an outer end of the covered background area to an outer endof the uncovered background area based on the foreground area;simultaneous-equation generating means for generating simultaneousequations consisting of a plurality of relational expressions by settingpixel values of pixels within the unit of processing determined based onthe unit of processing and by setting a divided known value obtained bydividing the foreground object components in the mixed area by a setnumber of divided portions; and calculation means for calculating theforeground object components in which the amount of motion blur isadjusted by solving the simultaneous equations.

[0017] By utilizing a characteristic in which the foreground objectcomponents contained in the first pixel from an end of the mixed areaare subtracted from the foreground object components contained in thesecond pixel, which is located adjacent to the first pixel, from the endof the mixed area on the straight line so that the single foregroundobject component corresponding to the second pixel is calculated, thecalculation means may sequentially solve the simultaneous equations fromthe relational expressions corresponding to the pixel located at theend, thereby calculating the foreground object components in which theamount of motion blur is adjusted.

[0018] The simultaneous-equation generating means may generate thesimultaneous equations based on the number of divided portions inaccordance with the amount of movement of the foreground object.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019]FIG. 1 illustrates the principle of the present invention.

[0020]FIG. 2 is a block diagram illustrating an example of theconfiguration to which the present invention is applied.

[0021]FIG. 3 is a block diagram illustrating an example of theconfiguration of a signal processor 12 shown in FIG. 2.

[0022]FIG. 4 is a block diagram illustrating the signal processor 12.

[0023]FIG. 5 illustrates the image capturing performed by a sensor.

[0024]FIG. 6 illustrates the arrangement of pixels.

[0025]FIG. 7 illustrates the operation of a detection device.

[0026]FIG. 8A illustrates an image obtained by image-capturing an objectcorresponding to a moving foreground and an object corresponding to astationary background.

[0027]FIG. 8B illustrates a model of an image obtained byimage-capturing an object corresponding to a moving foreground and anobject corresponding to a stationary background.

[0028]FIG. 9 illustrates a background area, a foreground area, a mixedarea, a covered background area, and an uncovered background area.

[0029]FIG. 10 illustrates a model obtained by expanding in the timedirection the pixel values of pixels aligned side-by-side in an imageobtained by image-capturing an object corresponding to a stationaryforeground and an the object corresponding to a stationary background.

[0030]FIG. 11 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0031]FIG. 12 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0032]FIG. 13 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0033]FIG. 14 illustrates an example in which pixels in a foregroundarea, a background area, and a mixed area are extracted.

[0034]FIG. 15 illustrates the relationships between pixels and a modelobtained by expanding the pixel values in the time direction.

[0035]FIG. 16 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0036]FIG. 17 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0037]FIG. 18 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0038]FIG. 19 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0039]FIG. 20 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0040]FIG. 21 is a flowchart illustrating the processing for adjustingthe amount of motion blur.

[0041]FIG. 22 is a block diagram illustrating an example of theconfiguration of an area specifying unit 103.

[0042]FIG. 23 illustrates an image when an object corresponding to aforeground is moving.

[0043]FIG. 24 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0044]FIG. 25 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0045]FIG. 26 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0046]FIG. 27 illustrates the conditions for determining the area.

[0047]FIG. 28A illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0048]FIG. 28B illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0049]FIG. 28C illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0050]FIG. 28D illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0051]FIG. 29 illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0052]FIG. 30 is a flowchart illustrating the area specifyingprocessing.

[0053]FIG. 31 is a block diagram illustrating an example of anotherconfiguration of the area specifying unit 103.

[0054]FIG. 32 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0055]FIG. 33 illustrates an example of a background image.

[0056]FIG. 34 is a block diagram illustrating the configuration of abinary-object-image extracting portion 302.

[0057]FIG. 35A illustrates the calculation of a correlation value.

[0058]FIG. 35B illustrates the calculation of a correlation value.

[0059]FIG. 36A illustrates the calculation of a correlation value.

[0060]FIG. 36B illustrates the calculation of a correlation value.

[0061]FIG. 37 illustrates an example of the binary object image.

[0062]FIG. 38 is a block diagram illustrating the configuration of atime change detector 303.

[0063]FIG. 39 illustrates determinations made by an area determiningportion 342.

[0064]FIG. 40 illustrates an example of determinations made by the timechange detector 303.

[0065]FIG. 41 is a flowchart illustrating the area specifying processingperformed by the area specifying unit 103.

[0066]FIG. 42 is a flowchart illustrating details of the area specifyingprocessing.

[0067]FIG. 43 is a block diagram illustrating still anotherconfiguration of the area specifying unit 103.

[0068]FIG. 44 is a block diagram illustrating the configuration of arobust-processing portion 361.

[0069]FIG. 45 illustrates motion compensation performed by a motioncompensator 381.

[0070]FIG. 46 illustrates motion compensation performed by the motioncompensator 381.

[0071]FIG. 47 is a flowchart illustrating the area specifyingprocessing.

[0072]FIG. 48 is a flowchart illustrating details of the robustprocessing.

[0073]FIG. 49 is a block diagram illustrating an example of theconfiguration of a mixture-ratio calculator 104.

[0074]FIG. 50 illustrates an example of the ideal mixture ratio α.

[0075]FIG. 51 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0076]FIG. 52 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0077]FIG. 53 illustrates the approximation using the correlation offoreground components.

[0078]FIG. 54 illustrates the relationship among C, N, and P.

[0079]FIG. 55 is a block diagram illustrating the configuration of anestimated-mixture-ratio processor 401.

[0080]FIG. 56 illustrates an example of the estimated mixture ratio.

[0081]FIG. 57 is a block diagram illustrating another configuration ofthe mixture-ratio calculator 104.

[0082]FIG. 58 is a flowchart illustrating the processing for calculatingthe mixture ratio.

[0083]FIG. 59 is a flowchart illustrating the processing for calculatingthe estimated mixture ratio.

[0084]FIG. 60 illustrates a straight line for approximating the mixtureratio α.

[0085]FIG. 61 illustrates a plane for approximating the mixture ratio α.

[0086]FIG. 62 illustrates the relationships of the pixels in a pluralityof frames when the mixture ratio α is calculated.

[0087]FIG. 63 is a block diagram illustrating another configuration ofthe mixture-ratio estimation processor 401.

[0088]FIG. 64 illustrates an example of the estimated mixture ratio.

[0089]FIG. 65 is a flowchart illustrating the mixture-ratio calculationprocessing.

[0090]FIG. 66 is a flowchart illustrating the mixture-ratio estimatingprocessing by using a model corresponding to a covered background area.

[0091]FIG. 67 is a block diagram illustrating an example of theconfiguration of a foreground/background separator 105.

[0092]FIG. 68A illustrates an input image, a foreground component image,and a background component image.

[0093]FIG. 68B illustrates a model of an input image, a foregroundcomponent image, and a background component image.

[0094]FIG. 69 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0095]FIG. 70 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0096]FIG. 71 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0097]FIG. 72 is a block diagram illustrating an example of theconfiguration of a separating portion 601.

[0098]FIG. 73A illustrates an example of a separated foregroundcomponent image.

[0099]FIG. 73B illustrates an example of a separated backgroundcomponent image.

[0100]FIG. 74 is a flowchart illustrating the processing for separatinga foreground and a background.

[0101]FIG. 75 is a block diagram illustrating an example of theconfiguration of a motion-blur adjusting unit 106.

[0102]FIG. 76 illustrates the unit of processing.

[0103]FIG. 77 illustrates a model indicating the relationship betweenpixel values and foreground components.

[0104]FIG. 78 illustrates the calculation of foreground components.

[0105]FIG. 79 illustrates the calculation of foreground components.

[0106]FIG. 80 illustrates a model in which the pixel values of aforeground component image are expanded in the time direction and theperiod corresponding to the shutter time is divided.

[0107]FIG. 81 is a flowchart illustrating the processing for adjustingmotion blur contained in a foreground.

[0108]FIG. 82 is a block diagram illustrating another configuration ofthe function of the signal processor 12.

[0109]FIG. 83 illustrates the configuration of a synthesizer 1001.

[0110]FIG. 84 is a block diagram illustrating still anotherconfiguration of the function of the signal processor 12.

[0111]FIG. 85 is a block diagram illustrating the configuration of amixture-ratio calculator 1101.

[0112]FIG. 86 is a block diagram illustrating the configuration of aforeground/background separator 1102.

BEST MODE FOR CARRYING OUT THE INVENTION

[0113]FIG. 1 illustrates the principle of the present invention. Asshown in FIG. 1, a first signal, which is information of a real society1 having a space and a time axis, is obtained by a sensor 2, and isformed into data. Data 3, which is the detection signal obtained by thesensor 2, is information obtained by projecting the information of thereal society 1 onto a time space having a dimension lower than the realsociety. Accordingly, the projected information has distortion caused bythe projection. In other words, the data 3 output from the sensor 2 hasdistortion with respect to the information of the real society 1.Although the data 3 has distortion caused by the projection, it containssignificant information for correcting for the distortion.

[0114] Accordingly, in the present invention, by performing signalprocessing on the data output from the sensor 2 by a signal processor 4,the distortion can be removed, reduced, or adjusted. Also, in thepresent invention, by performing signal processing on the data outputfrom the sensor 2 by the signal processor 4, significant information canbe extracted.

[0115]FIG. 2 illustrates an example of the configuration of a signalprocessing apparatus to which the present invention is applied. A sensor11, which is formed of, for example, a video camera, captures an imageof the real society, and outputs the obtained image data to a signalprocessor 12. The signal processor 12, which is formed of, for example,a personal computer, processes the data input from the sensor 11,adjusts the amount of distortion caused by the projection, specifies thearea in which significant information is embedded by the projection,extracts the significant information from the specified area, orprocesses the input data based on the extracted significant information.

[0116] The above-described significant information is, for example, themixture ratio, which is discussed below.

[0117] It can be considered that the information indicating the area inwhich the significant information embedded by the projection iscontained is also significant information. The area information, whichis described below, corresponds to the significant information.

[0118] The area in which the significant information is contained is,for example, a mixed area, which is discussed below.

[0119] The signal processor 12 is configured, for example, as shown inFIG. 3. A CPU (Central Processing Unit) 21 executes various types ofprocessing according to programs stored in a ROM (Read Only Memory) 22or in a storage unit 28. Programs executed by the CPU 21 and data arestored in a RAM (Random Access Memory) 23 as required. The CPU 21, theROM 22, and the RAM 23 are connected to each other by a bus 24.

[0120] An input/output interface 25 is also connected to the CPU 21 viathe bus 24. An input unit 26, which is formed of a keyboard, a mouse, amicrophone, and so on, and an output unit 27, which is formed of adisplay, a speaker, and so on, are connected to the input/outputinterface 25. The CPU 21 executes various types of processing inresponse to a command input from the input unit 26. The CPU 21 thenoutputs an image or sound obtained as a result of the processing to theoutput unit 27.

[0121] The storage unit 28 connected to the input/output interface 25 isformed of, for example, a hard disk, and stores programs executed by theCPU 21 and various types of data. A communication unit 29 communicateswith an external device via the Internet or another network. In thisexample, the communication unit 29 serves as an obtaining unit forobtaining an output of a sensor.

[0122] Alternatively, a program may be obtained via the communicationunit 29 and stored in the storage unit 28.

[0123] A drive 30 connected to the input/output interface 25 drives amagnetic disk 51, an optical disc 52, a magneto-optical disk 53, asemiconductor memory 54, or the like, when such a recording medium isattached to the drive 30, and obtains a program or data stored in thecorresponding medium. The obtained program or data is transferred to thestorage unit 28 and stored therein if necessary.

[0124] By taking a more specific example, a description is now given ofa signal processing apparatus which performs processing, such asspecifying an area having significant information embedded therein orextracting significant information embedded therein from data obtainedby a sensor. In the subsequent example, a CCD line sensor or a CCD areasensor corresponds to the sensor, the area information or the mixtureratio corresponds to the significant information, and the mixture stateof a foreground and a background or motion blur in a mixed areacorresponds to distortion.

[0125]FIG. 4 is a block diagram illustrating the signal processor 12.

[0126] It does not matter whether the individual functions of the signalprocessor 12 are implemented by hardware or software. That is, the blockdiagrams of this specification may be hardware block diagrams orsoftware functional block diagrams.

[0127] Motion blur is a distortion contained in an image correspondingto a moving object caused by the movement of an object to be captured inthe real world and the image-capturing characteristics of the sensor 11.

[0128] In this specification, an image to be captured corresponding toan object in the real world is referred to as an image object.

[0129] An input image supplied to the signal processor 12 is supplied toan object extracting unit 101, an area specifying unit 103, amixture-ratio calculator 104, and a foreground/background separator 105.

[0130] The object extracting unit 101 extracts a rough image objectcorresponding to a foreground object contained in the input image, andsupplies the extracted image object to a motion detector 102. The objectextracting unit 101 detects, for example, an outline of the foregroundimage object contained in the input image so as to extract a rough imageobject corresponding to the foreground object.

[0131] The object extracting unit 101 extracts a rough image objectcorresponding to a background object contained in the input image, andsupplies the extracted image object to the motion detector 102. Theobject extracting unit 101 extracts a rough image object correspondingto the background object from, for example, the difference between theinput image and the extracted image object corresponding to theforeground object.

[0132] Alternatively, for example, the object extracting unit 101 mayextract the rough image object corresponding to the foreground objectand the rough image object corresponding to the background object fromthe difference between the background image stored in a built-inbackground memory and the input image.

[0133] The motion detector 102 calculates a motion vector of the roughlyextracted image object corresponding to the foreground object accordingto a technique, such as block matching, gradient, phase correlation, orpel-recursive technique, and supplies the calculated motion vector andthe motion-vector positional information (which is information forspecifying the positions of the pixels corresponding to the motionvector) to the area specifying unit 103, the mixture-ratio calculator104, and a motion-blur extracting unit 106.

[0134] The motion vector output from the motion detector 102 containsinformation corresponding to the amount of movement v.

[0135] The motion detector 102 may output the motion vector of eachimage object, together with the pixel positional information forspecifying the pixels of the image object, to the motion-blur adjustingunit 106.

[0136] The amount of movement v is a value indicating a positionalchange in an image corresponding to a moving object in units of thepixel pitch. For example, if an object image corresponding to aforeground is moving such that it is displayed at a position four pixelsaway from a reference frame when it is positioned in the subsequentframe, the amount of movement v of the object image corresponding to theforeground is 4.

[0137] The object extracting unit 101 and the motion detector 102 areneeded when adjusting the amount of motion blur corresponding to amoving object.

[0138] The area specifying unit 103 determines to which of a foregroundarea, a background area, or a mixed area each pixel of the input imagebelongs, and supplies information indicating to which area each pixelbelongs (hereinafter referred to as “area information”) to themixture-ratio calculator 104, the foreground/background separator 105,and the motion-blur adjusting unit 106.

[0139] The mixture-ratio calculator 104 calculates the mixture ratiocorresponding to the pixels contained in a mixed area 63 (hereinafterreferred to as the “mixture ratio α”) based on the input image, and thearea information supplied from the area specifying unit 103, andsupplies the mixture ratio α to the foreground/background separator 105.

[0140] The mixture ratio α is a value indicating the ratio of the imagecomponents corresponding to the background object (hereinafter also bereferred to as “background components”) to the pixel value as expressedby equation (3), which is shown below.

[0141] The foreground/background separator 105 separates the input imageinto a foreground component image formed of only the image componentscorresponding to the foreground object (hereinafter also be referred toas “foreground components”) and a background component image formed ofonly the background components based on the area information suppliedfrom the area specifying unit 103 and the mixture ratio α supplied fromthe mixture-ratio calculator 104, and supplies the foreground componentimage to the motion-blur adjusting unit 106 and a selector 107. Theseparated foreground component image may be set as the final output. Amore precise foreground and background can be obtained compared to aknown method in which only a foreground and a background are specifiedwithout considering the mixed area.

[0142] The motion-blur adjusting unit 106 determines the unit ofprocessing indicating at least one pixel contained in the foregroundcomponent image based on the amount of movement v obtained from themotion vector and based on the area information. The unit of processingis data that specifies a group of pixels to be subjected to themotion-blur adjustments.

[0143] Based on the amount by which the motion blur is to be adjusted,which is input into the signal processor 12, the foreground componentimage supplied from the foreground/background separator 105, the motionvector and the positional information thereof supplied from the motiondetector 102, and the unit of processing, the motion-blur adjusting unit106 adjusts the amount of motion blur contained in the foregroundcomponent image by removing, decreasing, or increasing the motion blurcontained in the foreground component image. The motion-blur adjustingunit 106 then outputs the foreground component image in which amount ofmotion blur is adjusted to the selector 107. It is not essential thatthe motion vector and the positional information thereof be used.

[0144] The selector 107 selects one of the foreground component imagesupplied from the foreground/background separator 105 and the foregroundcomponent image in which the amount of motion blur is adjusted suppliedfrom the motion-blur adjusting unit 106 based on, for example, aselection signal reflecting a user's selection, and outputs the selectedforeground component image.

[0145] An input image supplied to the signal processor 12 is discussedbelow with reference to FIGS. 5 through 20.

[0146]FIG. 5 illustrates image capturing performed by a sensor. Thesensor 11 is formed of, for example, a CCD (Charge-Coupled Device) videocamera provided with a CCD area sensor, which is a solid-state imagingdevice. An object 111 corresponding to a foreground in the real worldmoves, for example, horizontally from the left to the right, between anobject 112 corresponding to a background and the sensor.

[0147] The sensor 11 captures the image of the object 111 correspondingto the foreground together with the image of the object 112corresponding to the background. The sensor 11 outputs the capturedimage in units of frames. For example, the sensor 11 outputs an imagehaving 30 frames per second. The exposure time of the sensor 11 can be1/30 second. The exposure time is a period from when the sensor 11starts converting input light into electrical charge until when theconversion from the input light to the electrical charge is finished.The exposure time is also referred to as a “shutter time”.

[0148]FIG. 6 illustrates the arrangement of pixels. In FIG. 6, A throughI indicate the individual pixels. The pixels are disposed on a plane ofa corresponding image. One detection device corresponding to each pixelis disposed on the sensor 11. When the sensor 11 performs imagecapturing, each detection device outputs a pixel value of thecorresponding pixel forming the image. For example, the position of thedetection device in the X direction corresponds to the horizontaldirection on the image, while the position of the detection device inthe Y direction corresponds to the vertical direction on the image.

[0149] As shown in FIG. 7, the detection device, which is, for example,a CCD, converts input light into electrical charge during a periodcorresponding to a shutter time, and stores the converted electricalcharge. The amount of charge is almost proportional to the intensity ofthe input light and the period for which the light is input. Thedetection device sequentially adds the electrical charge converted fromthe input light to the stored electrical charge during the periodcorresponding to the shutter time. That is, the detection deviceintegrates the input light during the period corresponding to theshutter time and stores the electrical charge corresponding to theamount of integrated light. It can be considered that the detectiondevice has an integrating function with respect to time.

[0150] The electrical charge stored in the detection device is convertedinto a voltage value by a circuit (not shown), and the voltage value isfurther converted into a pixel value, such as digital data, and isoutput. Accordingly, each pixel value output from the sensor 11 is avalue projected on a linear space, which is a result of integrating acertain three-dimensional portion of the object corresponding to theforeground or the background with respect to the shutter time.

[0151] The signal processor 12 extracts significant information embeddedin the output signal, for example, the mixture ratio α, by the storageoperation of the sensor 11. The signal processor 12 adjusts the amountof distortion, for example, the amount of motion blur, caused by themixture of the foreground image object itself. The signal processor 12also adjusts the amount of distortion caused by the mixture of theforeground image object and the background image object.

[0152]FIG. 8A illustrates an image obtained by capturing a moving objectcorresponding to a foreground and a stationary object corresponding to abackground. In the example shown in FIG. 8A, the object corresponding tothe foreground is moving horizontally from the left to the right withrespect to the screen.

[0153]FIG. 8B illustrates a model obtained by expanding pixel valuescorresponding to one line of the image shown in FIG. 8A in the timedirection. The horizontal direction shown in FIG. 8B corresponds to thespatial direction X in FIG. 8A.

[0154] The values of the pixels in the background area are formed onlyfrom the background components, that is, the image componentscorresponding to the background object. The values of the pixels in theforeground area are formed only from the foreground components, that is,the image components corresponding to the foreground object.

[0155] The values of the pixels of the mixed area are formed from thebackground components and the foreground components. Since the values ofthe pixels in the mixed area are formed from the background componentsand the foreground components, it may be referred to as a “distortionarea”. The mixed area is further classified into a covered backgroundarea and an uncovered background area.

[0156] The covered background area is a mixed area at a positioncorresponding to the leading end in the direction in which theforeground object is moving, where the background components aregradually covered with the foreground over time.

[0157] In contrast, the uncovered background area is a mixed areacorresponding to the trailing end in the direction in which theforeground object is moving, where the background components graduallyappear over time.

[0158] As discussed above, the image containing the foreground area, thebackground area, or the covered background area or the uncoveredbackground area is input into the area specifying unit 103, themixture-ratio calculator 104, and the foreground/background separator105 as the input image.

[0159]FIG. 9 illustrates the background area, the foreground area, themixed area, the covered background area, and the uncovered backgroundarea discussed above. In the areas corresponding to the image shown inFIG. 8B, the background area is a stationary portion, the foregroundarea is a moving portion, the covered background area of the mixed areais a portion that changes from the background to the foreground, and theuncovered background area of the mixed area is a portion that changesfrom the foreground to the background.

[0160]FIG. 10 illustrates a model obtained by expanding in the timedirection the pixel values of the pixels aligned side-by-side in theimage obtained by capturing the image of the object corresponding to thestationary foreground and the image of the object corresponding to thestationary background. For example, as the pixels aligned side-by-side,pixels arranged in one line on the screen can be selected.

[0161] The pixel values indicated by F01 through F04 shown in FIG. 10are values of the pixels corresponding to the object of the stationaryforeground. The pixel values indicated by B01 through B04 shown in FIG.10 are values of the pixels corresponding to the object of thestationary background.

[0162] Time elapses from the top to the bottom in FIG. 10 in thevertical direction in FIG. 10. The position at the top side of therectangle in FIG. 10 corresponds to the time at which the sensor 11starts converting input light into electrical charge, and the positionat the bottom side of the rectangle in FIG. 10 corresponds to the timeat which the conversion from the input light into the electrical chargeis finished. That is, the distance from the top side to the bottom sideof the rectangle in FIG. 10 corresponds to the shutter time.

[0163] The pixels shown in FIG. 10 are described below assuming that,for example, the shutter time is equal to the frame size.

[0164] The horizontal direction in FIG. 10 corresponds to the spatialdirection X in FIG. 8A. More specifically, in the example shown in FIG.10, the distance from the left side of the rectangle indicated by “F01”in FIG. 10 to the right side of the rectangle indicated by “B04” iseight times the pixel pitch, i.e., eight consecutive pixels.

[0165] When the foreground object and the background object arestationary, the light input into the sensor 11 does not change duringthe period corresponding to the shutter time.

[0166] The period corresponding to the shutter time is divided into twoor more portions of equal periods. For example, if the number of virtualdivided portions is 4, the model shown in FIG. 10 can be represented bythe model shown in FIG. 11. The number of virtual divided portions canbe set according to the amount of movement v of the object correspondingto the foreground within the shutter time. For example, the number ofvirtual divided portions is set to 4 when the amount of movement v is 4,and the period corresponding to the shutter time is divided into fourportions.

[0167] The uppermost line in FIG. 11 corresponds to the first dividedperiod from when the shutter has opened. The second line in FIG. 11corresponds to the second divided period from when the shutter hasopened. The third line in FIG. 11 corresponds to the third dividedperiod from when the shutter has opened. The fourth line in FIG. 11corresponds to the fourth divided period from when the shutter hasopened.

[0168] The shutter time divided in accordance with the amount ofmovement v is also hereinafter referred to as the “shutter time/v”.

[0169] When the object corresponding to the foreground is stationary,the light input into the sensor 11 does not change, and thus, theforeground component F01/v is equal to the value obtained by dividingthe pixel value F01 by the number of virtual divided portions.Similarly, when the object corresponding to the foreground isstationary, the foreground component F02/v is equal to the valueobtained by dividing the pixel value F02 by the number of virtualdivided portions, the foreground component F03/v is equal to the valueobtained by dividing the pixel value F03 by the number of virtualdivided portions, and the foreground component F04/v is equal to thevalue obtained by dividing the pixel value F04 by the number of virtualdivided portions.

[0170] When the object corresponding to the background is stationary,the light input into the sensor 11 does not change, and thus, thebackground component B01/v is equal to the value obtained by dividingthe pixel value B01 by the number of virtual divided portions.Similarly, when the object corresponding to the background isstationary, the background component B02/v is equal to the valueobtained by dividing the pixel value B02 by the number of virtualdivided portions, the background component B03/v is equal to the valueobtained by dividing the pixel value B03 by the number of virtualdivided portions, and the background component B04/v is equal to thevalue obtained by dividing the pixel value B04 by the number of virtualdivided portions.

[0171] More specifically, when the object corresponding to theforeground is stationary, the light corresponding to the foregroundobject input into the sensor 11 does not change during the periodcorresponding to the shutter time. Accordingly, the foreground componentF01/v corresponding to the first portion of the shutter time/v from whenthe shutter has opened, the foreground component F01/v corresponding tothe second portion of the shutter time/v from when the shutter hasopened, the foreground component F01/v corresponding to the thirdportion of the shutter time/v from when the shutter has opened, and theforeground component F01/v corresponding to the fourth portion of theshutter time/v from when the shutter has opened become the same value.The same applies to F02/v through F04/v, as in the case of F01/v.

[0172] When the object corresponding to the background is stationary,the light corresponding to the background object input into the sensor11 does not change during the period corresponding to the shutter time.Accordingly, the background component B01/v corresponding to the firstportion of the shutter time/v from when the shutter has opened, thebackground component B01/v corresponding to the second portion of theshutter time/v from when the shutter has opened, the backgroundcomponent B01/v corresponding to the third portion of the shutter time/vfrom when the shutter has opened, and the background component B01/vcorresponding to the fourth portion of the shutter time/v from when theshutter has opened become the same value. The same applies to B02/vthrough B04/v.

[0173] A description is given of the case in which the objectcorresponding to the foreground is moving and the object correspondingto the background is stationary.

[0174]FIG. 12 illustrates a model obtained by expanding in the timedirection the pixel values of the pixels in one line, including acovered background area, when the object corresponding to the foregroundis moving to the right in FIG. 12. In FIG. 12, the amount of movement vis 4. Since one frame is a short period, it can be assumed that theobject corresponding to the foreground is a rigid body moving withconstant velocity. In FIG. 12, the object image corresponding to theforeground is moving such that it is positioned four pixels to the rightwith respect to a reference frame when it is displayed in the subsequentframe.

[0175] In FIG. 12, the pixels from the leftmost pixel to the fourthpixel belong to the foreground area. In FIG. 12, the pixels from thefifth pixel to the seventh pixel from the left belong to the mixed area,which is the covered background area. In FIG. 12, the rightmost pixelbelongs to the background area.

[0176] The object corresponding to the foreground is moving such that itgradually covers the object corresponding to the background over time.Accordingly, the components contained in the pixel values of the pixelsbelonging to the covered background area change from the backgroundcomponents to the foreground components at a certain time during theperiod corresponding to the shutter time.

[0177] For example, the pixel value M surrounded by the thick frame inFIG. 12 is expressed by equation (1) below.

M=B02/v+B02/v+F07/v+F06/v  (1)

[0178] For example, the fifth pixel from the left contains a backgroundcomponent corresponding to one portion of the shutter time/v andforeground components corresponding to three portions of the shuttertime/v, and thus, the mixture ratio α of the fifth pixel from the leftis 1/4. The sixth pixel from the left contains background componentscorresponding to two portions of the shutter time/v and foregroundcomponents corresponding to two portions of the shutter time/v, andthus, the mixture ratio α of the sixth pixel from the left is 1/2. Theseventh pixel from the left contains background components correspondingto three portions of the shutter time/v and a foreground componentcorresponding to one portion of the shutter time/v, and thus, themixture ratio α of the fifth pixel from the left is 3/4.

[0179] It can be assumed that the object corresponding to the foregroundis a rigid body, and the foreground object is moving with constantvelocity such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, for example, the foreground componentF07/v of the fourth pixel from the left in FIG. 12 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the fifth pixel from the left inFIG. 12 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F07/vis equal to the foreground component of the sixth pixel from the left inFIG. 12 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened, and the foreground component of the seventhpixel from the left in FIG. 12 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened.

[0180] It can be assumed that the object corresponding to the foregroundis a rigid body, and the foreground object is moving with constantvelocity such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, for example, the foreground componentF06/v of the third pixel from the left in FIG. 12 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the fourth pixel from the left inFIG. 12 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F06/vis equal to the foreground component of the fifth pixel from the left inFIG. 12 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened, and the foreground component of the sixthpixel from the left in FIG. 12 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened.

[0181] It can be assumed that the object corresponding to the foregroundis a rigid body, and the foreground object is moving with constantvelocity such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, for example, the foreground componentF05/v of the second pixel from the left in FIG. 12 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the third pixel from the left inFIG. 12 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F05/vis equal to the foreground component of the fourth pixel from the leftin FIG. 12 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened, and the foreground component of the fifthpixel from the left in FIG. 12 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened.

[0182] It can be assumed that the object corresponding to the foregroundis a rigid body, and the foreground object is moving with constantvelocity such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, for example, the foreground componentF04/v of the left most pixel in FIG. 12 corresponding to the firstportion of the shutter time/v from when the shutter has opened is equalto the foreground component of the second pixel from the left in FIG. 12corresponding to the second portion of the shutter time/v from when theshutter has opened. Similarly, the foreground component F04/v is equalto the foreground component of the third pixel from the left in FIG. 12corresponding to the third portion of the shutter time/v from when theshutter has opened, and the foreground component of the fourth pixelfrom the left in FIG. 12 corresponding to the fourth portion of theshutter time/v from when the shutter has opened.

[0183] Since the foreground area corresponding to the moving objectcontains motion blur as discussed above, it can also be referred to as a“distortion area”.

[0184]FIG. 13 illustrates a model obtained by expanding in the timedirection the pixel values of the pixels in one line including anuncovered background area when the object corresponding to theforeground is moving to the right in FIG. 13. In FIG. 13, the amount ofmovement v is 4. Since one frame is a short period, it can be assumedthat the object corresponding to the foreground is a rigid body movingwith constant velocity. In FIG. 13, the object image corresponding tothe foreground is moving to the right such that it is positioned fourpixels to the right with respect to a reference frame when it isdisplayed in the subsequent frame.

[0185] In FIG. 13, the pixels from the leftmost pixel to the fourthpixel belong to the background area. In FIG. 13, the pixels from thefifth pixel to the seventh pixels from the left belong to the mixedarea, which is an uncovered background area. In FIG. 13, the rightmostpixel belongs to the foreground area.

[0186] The object corresponding to the foreground which covers theobject corresponding to the background is moving such that it isgradually removed from the object corresponding to the background overtime. Accordingly, the components contained in the pixel values of thepixels belonging to the uncovered background area change from theforeground components to the background components at a certain time ofthe period corresponding to the shutter time.

[0187] For example, the pixel value M′ surrounded by the thick frame inFIG. 13 is expressed by equation (2).

M′=F02/v+F01/v+B26/v+B26/v  (2)

[0188] For example, the fifth pixel from the left contains backgroundcomponents corresponding to three portions of the shutter time/v and aforeground component corresponding to one shutter portion of the shuttertime/v, and thus, the mixture ratio α of the fifth pixel from the leftis 3/4. The sixth pixel from the left contains background componentscorresponding to two portions of the shutter time/v and foregroundcomponents corresponding to two portions of the shutter time/v, andthus, the mixture ratio α of the sixth pixel from the left is 1/2. Theseventh pixel from the left contains a background componentcorresponding to one portion of the shutter time/v and foregroundcomponents corresponding to three portions of the shutter time/v, andthus, the mixture ratio α of the seventh pixel from the left is 1/4.

[0189] When equations (1) and (2) are generalized, the pixel value M canbe expressed by equation (3): $\begin{matrix}{M = {{\alpha \cdot B} + {\sum\limits_{i}^{\quad}{{Fi}/v}}}} & (3)\end{matrix}$

[0190] where α is the mixture ratio, B indicates a pixel value of thebackground, and Fi/v designates a foreground component.

[0191] It can be assumed that the object corresponding to the foregroundis a rigid body, which is moving with constant velocity, and the amountof movement is 4. Accordingly, for example, the foreground componentF01/v of the fifth pixel from the left in FIG. 13 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the sixth pixel from the left inFIG. 13 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F01/vis equal to the foreground component of the seventh pixel from the leftin FIG. 13 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened, and the foreground component of the eighthpixel from the left in FIG. 13 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened.

[0192] It can be assumed that the object corresponding to the foregroundis a rigid body, which is moving with constant velocity, and the amountof movement v is 4. Accordingly, for example, the foreground componentF02/v of the sixth pixel from the left in FIG. 13 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the seventh pixel from the left inFIG. 13 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F02/vis equal to the foreground component of the eighth pixel from the leftin FIG. 13 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened.

[0193] It can be assumed that the object corresponding to the foregroundis a rigid body, which is moving with constant velocity, and the amountof movement v is 4. Accordingly, for example, the foreground componentF03/v of the seventh pixel from the left in FIG. 13 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the eighth pixel from the left inFIG. 13 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened.

[0194] It has been described with reference to FIGS. 11 through 13 thatthe number of virtual divided portions is 4. The number of virtualdivided portions corresponds to the amount of movement v. Generally, theamount of movement v corresponds to the moving speed of the objectcorresponding to the foreground. For example, if the objectcorresponding to the foreground is moving such that it is displayed fourpixels to the right with respect to a certain frame when it ispositioned in the subsequent frame, the amount of movement v is set to4. The number of virtual divided portions is set to 4 in accordance withthe amount of movement v. Similarly, when the object corresponding tothe foreground is moving such that it is displayed six pixels to theleft with respect to a certain frame when it is positioned in thesubsequent frame, the amount of movement v is set to 6, and the numberof virtual divided portions is set to 6.

[0195]FIGS. 14 and 15 illustrate the relationship of the foregroundarea, the background area, and the mixed area which consists of acovered background or an uncovered background, which are discussedabove, to the foreground components and the background componentscorresponding to the divided periods of the shutter time.

[0196]FIG. 14 illustrates an example in which pixels in the foregroundarea, the background area, and the mixed area are extracted from animage containing a foreground corresponding to an object moving in frontof a stationary background. In the example shown in FIG. 14, the objectcorresponding to the foreground is horizontally moving with respect tothe screen.

[0197] Frame #n+1 is a frame subsequent to frame #n, and frame #n+2 is aframe subsequent to frame #n+1.

[0198] Pixels in the foreground area, the background area, and the mixedarea are extracted from one of frames #n through #n+2, and the amount ofmovement v is set to 4. A model obtained by expanding the pixel valuesof the extracted pixels in the time direction is shown in FIG. 15.

[0199] Since the object corresponding to the foreground is moving, thepixel values in the foreground area are formed of four differentforeground components corresponding to the shutter time/v. For example,the leftmost pixel of the pixels in the foreground area shown in FIG. 15consists of F01/v, F02/v, F03/v, and F04/v. That is, the pixels in theforeground contain motion blur.

[0200] Since the object corresponding to the background is stationary,light input into the sensor 11 corresponding to the background duringthe shutter time does not change. In this case, the pixel values in thebackground area do not contain motion blur.

[0201] The pixel values in the mixed area consisting of a coveredbackground area or an uncovered background area are formed of foregroundcomponents and background components.

[0202] A description is given below of a model obtained by expanding inthe time direction the pixel values of the pixels which are alignedside-by-side in a plurality of frames and which are located at the samepositions when the frames are overlapped when the image corresponding tothe object is moving. For example, when the image corresponding to theobject is moving horizontally with respect to the screen, pixels alignedon the screen can be selected as the pixels aligned side-by-side.

[0203]FIG. 16 illustrates a model obtained by expanding in the timedirection the pixels which are aligned side-by-side in three frames ofan image obtained by capturing an object corresponding to a stationarybackground and which are located at the same positions when the framesare overlapped. Frame #n is the frame subsequent to frame #n−1, andframe #n+1 is the frame subsequent to frame #n. The same applies to theother frames.

[0204] The pixel values B01 through B12 shown in FIG. 16 are pixelvalues corresponding to the stationary background object. Since theobject corresponding to the background is stationary, the pixel valuesof the corresponding pixels in frame #n−1 through frame #n+1 do notchange. For example, the pixel in frame #n and the pixel in frame #n+1located at the corresponding position of the pixel having the pixelvalue B05 in frame #n−1 have the pixel value B05.

[0205]FIG. 17 illustrates a model obtained by expanding in the timedirection the pixels which are aligned side-by-side in three frames ofan image obtained by capturing an object corresponding to a foregroundthat is moving to the right in FIG. 17 together with an objectcorresponding to a stationary background and which are located at thesame positions when the frames are overlapped. The model shown in FIG.17 contains a covered background area.

[0206] In FIG. 17, it can be assumed that the object corresponding tothe foreground is a rigid body moving with constant velocity, and thatit is moving such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, the amount of movement v is 4, and thenumber of virtual divided portions is 4.

[0207] For example, the foreground component of the leftmost pixel offrame #n−1 in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F12/v, and the foregroundcomponent of the second pixel from the left in FIG. 17 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F12/v. The foreground component of the third pixel fromthe left in FIG. 17 corresponding to the third portion of the shuttertime/v from when the shutter has opened and the foreground component ofthe fourth pixel from the left in FIG. 17 corresponding to the fourthportion of the shutter time/v from when the shutter has opened areF12/v.

[0208] The foreground component of the leftmost pixel of frame #n−1 inFIG. 17 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened is F11/v. The foreground component of thesecond pixel from the left in FIG. 17 corresponding to the third portionof the shutter time/v from when the shutter has opened is also F11/v.The foreground component of the third pixel from the left in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened is F11/v.

[0209] The foreground component of the leftmost pixel of frame #n−1 inFIG. 17 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened is F10/v. The foreground component of thesecond pixel from the left in FIG. 17 corresponding to the fourthportion of the shutter time/v from when the shutter has opened is alsoF10/v. The foreground component of the leftmost pixel of frame #n−1 inFIG. 17 corresponding to the fourth portion of the shutter time/v fromwhen the shutter has opened is F09/v.

[0210] Since the object corresponding to the background is stationary,the background component of the second pixel from the left of frame #n−1in FIG. 17 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is B01/v. The background components of thethird pixel from the left of frame #n−1 in FIG. 17 corresponding to thefirst and second portions of the shutter time/v from when the shutterhas opened are B02/v. The background components of the fourth pixel fromthe left of frame #n−1 in FIG. 17 corresponding to the first throughthird portions of the shutter time/v from when the shutter has openedare B03/v.

[0211] In frame #n−1 in FIG. 17, the leftmost pixel from the leftbelongs to the foreground area, and the second through fourth pixelsfrom the left belong to the mixed area, which is a covered backgroundarea.

[0212] The fifth through twelfth pixels from the left of frame #n−1 inFIG. 17 belong to the background area, and the pixel values thereof areB04 through B11, respectively.

[0213] The first through fifth pixels from the left in frame #n in FIG.17 belong to the foreground area. The foreground component in theshutter time/v in the foreground area of frame #n is any one of F05/vthrough F12/v.

[0214] It can be assumed that the object corresponding to the foregroundis a rigid body moving with constant velocity, and that it is movingsuch that the foreground image is displayed four pixels to the right inthe subsequent frame. Accordingly, the foreground component of the fifthpixel from the left of frame #n in FIG. 17 corresponding to the firstportion of the shutter time/v from when the shutter has opened is F12/v,and the foreground component of the sixth pixel from the left in FIG. 17corresponding to the second portion of the shutter time/v from when theshutter has opened is also F12/v. The foreground component of theseventh pixel from the left in FIG. 17 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the eighth pixel from the left in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F12/v.

[0215] The foreground component of the fifth pixel from the left offrame #n in FIG. 17 corresponding to the second portion of the shuttertime/v from when the shutter has opened is F11/v. The foregroundcomponent of the sixth pixel from the left in FIG. 17 corresponding tothe third portion of the shutter time/v from when the shutter has openedis also F11/v. The foreground component of the seventh pixel from theleft in FIG. 17 corresponding to the fourth portion of the shuttertime/v from when the shutter has opened is F11/v.

[0216] The foreground component of the fifth pixel from the left offrame #n in FIG. 17 corresponding to the third portion of the shuttertime/v from when the shutter has opened is F10/v. The foregroundcomponent of the sixth pixel from the left in FIG. 17 corresponding tothe fourth portion of the shutter time/v from when the shutter hasopened is also F10/v. The foreground component of the fifth pixel fromthe left of frame #n in FIG. 17 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened is F09/v.

[0217] Since the object corresponding to the background is stationary,the background component of the sixth pixel from the left of frame #n inFIG. 17 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is B05/v. The background components of theseventh pixel from the left of frame #n in FIG. 17 corresponding to thefirst and second portions of the shutter time/v from when the shutterhas opened are B06/v. The background components of the eighth pixel fromthe left of frame #n in FIG. 17 corresponding to the first through thirdportion of the shutter time/v from when the shutter has opened areB07/v.

[0218] In frame #n in FIG. 17, the sixth through eighth pixels from theleft belong to the mixed area, which is a covered background area.

[0219] The ninth through twelfth pixels from the left of frame #n inFIG. 17 belong to the background area, and the pixel values thereof areB08 through B11, respectively.

[0220] The first through ninth pixels from the left in frame #n+1 inFIG. 17 belong to the foreground area. The foreground component in theshutter time/v in the foreground area of frame #n+1 is any one of F01/vthrough F12/v.

[0221] It can be assumed that the object corresponding to the foregroundis a rigid body moving with constant velocity, and that it is movingsuch that the foreground image is displayed four pixels to the right inthe subsequent frame. Accordingly, the foreground component of the ninthpixel from the left of frame #n+1 in FIG. 17 corresponding to the firstportion of the shutter time/v from when the shutter has opened is F12/v,and the foreground component of the tenth pixel from the left in FIG. 17corresponding to the second portion of the shutter time/v from when theshutter has opened is also F12/v. The foreground component of theeleventh pixel from the left in FIG. 17 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the twelfth pixel from the left in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F12/v.

[0222] The foreground component of the ninth pixel from the left offrame #n+1 in FIG. 17 corresponding to the second portion of the shuttertime/v from when the shutter has opened is F11/v. The foregroundcomponent of the tenth pixel from the left in FIG. 17 corresponding tothe third portion of the shutter time/v from when the shutter has openedis also F11/v. The foreground component of the eleventh pixel from theleft in FIG. 17 corresponding to the fourth portion of the shuttertime/v from when the shutter has opened is F11/v.

[0223] The foreground component of the ninth pixel from the left offrame #n+1 in FIG. 17 corresponding to the third portion of the shuttertime/v from when the shutter has opened is F10/v. The foregroundcomponent of the tenth pixel from the left in FIG. 17 corresponding tothe fourth portion of the shutter time/v from when the shutter hasopened is also F10/v. The foreground component of the ninth pixel fromthe left of frame #n+1 in FIG. 17 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened is F09/v.

[0224] Since the object corresponding to the background is stationary,the background component of the tenth pixel from the left of frame #n+1in FIG. 17 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is B09/v. The background components of theeleventh pixel from the left of frame #n+1 in FIG. 17 corresponding tothe first and second portions of the shutter time/v from when theshutter has opened are B10/v. The background components of the twelfthpixel from the left of frame #n+1 in FIG. 17 corresponding to the firstthrough third portion of the shutter time/v from when the shutter hasopened are B11/v.

[0225] In frame #n+1 in FIG. 17, the tenth through twelfth pixels fromthe left belong to the mixed area, which is a covered background area.

[0226]FIG. 18 is a model of an image obtained by extracting theforeground components from the pixel values shown in FIG. 17.

[0227]FIG. 19 illustrates a model obtained by expanding in the timedirection the pixels which are aligned side-by-side in three frames ofan image obtained by capturing an object corresponding to a foregroundthat is moving to the right in FIG. 19 together with an objectcorresponding to a stationary background and which are located at thesame positions when the frames are overlapped. The model shown in FIG.19 contains an uncovered background area.

[0228] In FIG. 19, it can be assumed that the object corresponding tothe foreground is a rigid body moving with constant velocity, and thatit is moving such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, the amount of movement v is 4.

[0229] For example, the foreground component of the leftmost pixel offrame #n−1 in FIG. 19 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F13/v, and the foregroundcomponent of the second pixel from the left in FIG. 19 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F13/v. The foreground component of the third pixel fromthe left in FIG. 19 corresponding to the third portion of the shuttertime/v from when the shutter has opened and the foreground component ofthe fourth pixel from the left in FIG. 19 corresponding to the fourthportion of the shutter time/v from when the shutter has opened areF13/v.

[0230] The foreground component of the second pixel from the left offrame #n−1 in FIG. 19 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F14/v. The foregroundcomponent of the third pixel from the left in FIG. 19 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F14/v. The foreground component of the third pixel fromthe left in FIG. 19 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F15/v.

[0231] Since the object corresponding to the background is stationary,the background components of the leftmost pixel of frame #n−1 in FIG. 19corresponding to the second through fourth portions of the shuttertime/v from when the shutter has opened are B25/v. The backgroundcomponents of the second pixel from the left of frame #n−1 in FIG. 19corresponding to the third and fourth portions of the shutter time/vfrom when the shutter has opened are B26/v. The background component ofthe third pixel from the left of frame #n−1 in FIG. 19 corresponding tothe fourth portion of the shutter time/v from when the shutter hasopened is B27/v.

[0232] In frame #n−1 in FIG. 19, the leftmost pixel through the thirdpixel belong to the mixed area, which is an uncovered background area.

[0233] The fourth through twelfth pixels from the left of frame #n−1 inFIG. 19 belong to the foreground area. The foreground component of theframe is any one of F13/v through F24/v.

[0234] The leftmost pixel through the fourth pixel from the left offrame #n in FIG. 19 belong to the background area, and the pixel valuesthereof are B25 through B28, respectively.

[0235] It can be assumed that the object corresponding to the foregroundis a rigid body moving with constant velocity, and that it is movingsuch that it is displayed four pixels to the right in the subsequentframe. Accordingly, the foreground component of the fifth pixel from theleft of frame #n in FIG. 19 corresponding to the first portion of theshutter time/v from when the shutter has opened is F13/v, and theforeground component of the sixth pixel from the left in FIG. 19corresponding to the second portion of the shutter time/v from when theshutter has opened is also F13/v. The foreground component of theseventh pixel from the left in FIG. 19 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the eighth pixel from the left in FIG. 19corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F13/v.

[0236] The foreground component of the sixth pixel from the left offrame #n in FIG. 19 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F14/v. The foregroundcomponent of the seventh pixel from the left in FIG. 19 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F14/v. The foreground component of the eighth pixel fromthe left in FIG. 19 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F15/v.

[0237] Since the object corresponding to the background is stationary,the background components of the fifth pixel from the left of frame #nin FIG. 19 corresponding to the second through fourth portions of theshutter time/v from when the shutter has opened are B29/v. Thebackground components of the sixth pixel from the left of frame #n inFIG. 19 corresponding to the third and fourth portions of the shuttertime/v from when the shutter has opened are B30/v. The backgroundcomponent of the seventh pixel from the left of frame #n in FIG. 19corresponding to the fourth portion of the shutter time/v from when theshutter has opened is B31/v.

[0238] In frame #n in FIG. 19, the fifth pixel through the seventh pixelfrom the left belong to the mixed area, which is an uncovered backgroundarea.

[0239] The eighth through twelfth pixels from the left of frame #n inFIG. 19 belong to the foreground area. The value in the foreground areaof frame #n corresponding to the period of the shutter time/v is any oneof F13/v through F20/v.

[0240] The leftmost pixel through the eighth pixel from the left offrame #n+1 in FIG. 19 belong to the background area, and the pixelvalues thereof are B25 through B32, respectively.

[0241] It can be assumed that the object corresponding to the foregroundis a rigid body moving with constant velocity, and that it is movingsuch that it is displayed four pixels to the right in the subsequentframe. Accordingly, the foreground component of the ninth pixel from theleft of frame #n+1 in FIG. 19 corresponding to the first portion of theshutter time/v from when the shutter has opened is F13/v, and theforeground component of the tenth pixel from the left in FIG. 19corresponding to the second portion of the shutter time/v from when theshutter has opened is also F13/v. The foreground component of theeleventh pixel from the left in FIG. 19 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the twelfth pixel from the left in FIG. 19corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F13/v.

[0242] The foreground component of the tenth pixel from the left offrame #n+1 in FIG. 19 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F14/v. The foregroundcomponent of the eleventh pixel from the left in FIG. 19 correspondingto the second portion of the shutter time/v from when the shutter hasopened is also F14/v. The foreground component of the twelfth pixel fromthe left in FIG. 19 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F15/v.

[0243] Since the object corresponding to the background is stationary,the background components of the ninth pixel from the left of frame #n+1in FIG. 19 corresponding to the second through fourth portions of theshutter time/v from when the shutter has opened are B33/v. Thebackground components of the tenth pixel from the left of frame #n+1 inFIG. 19 corresponding to the third and fourth portions of the shuttertime/v from when the shutter has opened are B34/v. The backgroundcomponent of the eleventh pixel from the left of frame #n+1 in FIG. 19corresponding to the fourth portion of the shutter time/v from when theshutter has opened is B35/v.

[0244] In frame #n+1 in FIG. 19, the ninth through eleventh pixels fromthe left in FIG. 19 belong to the mixed area, which is an uncoveredbackground area.

[0245] The twelfth pixel from the left of frame #n+1 in FIG. 19 belongsto the foreground area. The foreground component in the shutter time/vin the foreground area of frame #n+1 is any one of F13 through F16,respectively.

[0246]FIG. 20 illustrates a model of an image obtained by extracting theforeground components from the pixel values shown in FIG. 19.

[0247] Referring back to FIG. 4, the area specifying unit 103 specifiesflags indicating to which of a foreground area, a background area, acovered background area, or an uncovered background area the individualpixels of the input image belong by using the pixel values of aplurality of frames, and supplies the flags to the mixture-ratiocalculator 104 and the motion-blur adjusting unit 106 as the areainformation.

[0248] The mixture-ratio calculator 104 calculates the mixture ratio αfor each pixel contained in the mixed area based on the pixel values ofa plurality of frames and the area information, and supplies thecalculated mixture ratio α to the foreground/background separator 105.

[0249] The foreground/background separator 105 extracts the foregroundcomponent image consisting of only the foreground components based onthe pixel values of a plurality of frames, the area information, and themixture ratio α, and supplies the foreground component image to themotion-blur adjusting unit 106.

[0250] The motion-blur adjusting unit 106 adjusts the amount of motionblur contained in the foreground component image based on the foregroundcomponent image supplied from the foreground/background separator 105,the motion vector supplied from the motion detector 102, and the areainformation supplied from the area specifying unit 103, and then outputsthe foreground component image in which motion blur is adjusted.

[0251] The processing for adjusting the amount of motion blur performedby the signal processor 12 is described below with reference to theflowchart of FIG. 21. In step S11, the area specifying unit 103 executesarea specifying processing, based on an input image, for generating areainformation indicating to which of a foreground area, a background area,a covered background area, or an uncovered background area each pixel ofthe input image belongs. Details of the area specifying processing aregiven below. The area specifying unit 103 supplies the generated areainformation to the mixture-ratio calculator 104.

[0252] In step S11, the area specifying unit 103 may generate, based onthe input image, area information indicating to which of the foregroundarea, the background area, or the mixed area (regardless of whether eachpixel belongs to a covered background area or an uncovered backgroundarea) each pixel of the input image belongs. In this case, theforeground/background separator 105 and the motion-blur adjusting unit106 determine based on the direction of the motion vector whether themixed area is a covered background area or an uncovered background area.For example, if the input image is disposed in the order of theforeground area, the mixed area, and the background area in thedirection of the motion vector, it is determined that the mixed area isa covered background area. If the input image is disposed in the orderof the background area, the mixed area, and the foreground area in thedirection of the motion vector, it is determined that the mixed area isan uncovered background area.

[0253] In step S12, the mixture-ratio calculator 104 calculates themixture ratio α for each pixel contained in the mixed area based on theinput image and the area information. Details of the mixture ratiocalculating processing are given below. The mixture-ratio calculator 104supplies the calculated mixture ratio α to the foreground/backgroundseparator 105.

[0254] In step S13, the foreground/background separator 105 extracts theforeground components from the input image based on the area informationand the mixture ratio α, and supplies the foreground components to themotion-blur adjusting unit 106 as the foreground component image.

[0255] In step S14, the motion-blur adjusting unit 106 generates, basedon the motion vector and the area information, the unit of processingthat indicates the positions of consecutive pixels disposed in themoving direction and belonging to any of the uncovered background area,the foreground area, and the covered background area, and adjusts theamount of motion blur contained in the foreground componentscorresponding to the unit of processing. Details of the processing foradjusting the amount of motion blur are given below.

[0256] In step S15, the signal processor 12 determines whether theprocessing is finished for the whole screen. If it is determined thatthe processing is not finished for the whole screen, the processproceeds to step S14, and the processing for adjusting the amount ofmotion blur for the foreground components corresponding to the unit ofprocessing is repeated.

[0257] If it is determined in step S15 that the processing is finishedfor the whole screen, the processing is completed.

[0258] In this manner, the signal processor 12 is capable of adjustingthe amount of motion blur contained in the foreground by separating theforeground and the background. That is, the signal processor 12 iscapable of adjusting the amount of motion blur contained in sampled dataindicating the pixel values of the foreground pixels.

[0259] The configuration of each of the area specifying unit 103, themixture-ratio calculator 104, the foreground/background separator 105,and the motion-blur adjusting unit 106 is described below.

[0260]FIG. 22 is a block diagram illustrating an example of theconfiguration of the area specifying unit 103. The area specifying unit103 shown in FIG. 22 does not use a motion vector. A frame memory 201stores an input image in units of frames. When the image to be processedis frame #n, the frame memory 201 stores frame #n−2, which is the frametwo frames before frame #n, frame #n−1, which is the frame one framebefore frame #n, frame #n, frame #n+1, which is the frame one frameafter frame #n, frame #n+2, which is the frame two frames after frame#n.

[0261] A stationary/moving determining portion 202-1 reads the pixelvalue of the pixel of frame #n+2 located at the same position as aspecific pixel of frame #n in which the area to which the pixel belongsis determined, and reads the pixel value of the pixel of frame #n+1located at the same position of the specific pixel of frame #n from theframe memory 201, and calculates the absolute value of the differencebetween the read pixel values. The stationary/moving determining portion202-1 determines whether the absolute value of the difference betweenthe pixel value of frame #n+2 and the pixel value of frame #n+l isgreater than a preset threshold Th. If it is determined that thedifference is greater than the threshold Th, a stationary/movingdetermination indicating “moving” is supplied to an area determiningportion 203-1. If it is determined that the absolute value of thedifference between the pixel value of the pixel of frame #n+2 and thepixel value of the pixel of frame #n+1 is smaller than or equal to thethreshold Th, the stationary/moving determining portion 202-1 supplies astationary/moving determination indicating “stationary” to the areadetermining portion 203-1.

[0262] A stationary/moving determining portion 202-2 reads the pixelvalue of a specific pixel of frame #n in which the area to which thepixel belongs is determined, and reads the pixel value of the pixel offrame #n+1 located at the same position as the specific pixel of frame#n from the frame memory 201, and calculates the absolute value of thedifference between the pixel values. The stationary/moving determiningportion 202-2 determines whether the absolute value of the differencebetween the pixel value of frame #n+1 and the pixel value of frame #n isgreater than a preset threshold Th. If it is determined that theabsolute value of the-difference between the pixel values is greaterthan the threshold Th, a stationary/moving determination indicating“moving” is supplied to the area determining portion 203-1 and an areadetermining portion 203-2. If it is determined that the absolute valueof the difference between the pixel value of the pixel of frame #n+1 andthe pixel value of the pixel of frame #n is smaller than or equal to thethreshold Th, the stationary/moving determining portion 202-2 supplies astationary/moving determination indicating “stationary” to the areadetermining portion 203-1 and the area determining portion 203-2.

[0263] A stationary/moving determining portion 202-3 reads the pixelvalue of a specific pixel of frame #n in which the area to which thepixel belongs is determined, and reads the pixel value of the pixel offrame #n−1 located at the same position as the specific pixel of frame#n from the frame memory 201, and calculates the absolute value of thedifference between the pixel values. The stationary/moving determiningportion 202-3 determines whether the absolute value of the differencebetween the pixel value of frame #n and the pixel value of frame #n−1 isgreater than a preset threshold Th. If it is determined that theabsolute value of the difference between the pixel values is greaterthan the threshold Th, a stationary/moving determination indicating“moving” is supplied to the area determining portion 203-2 and an areadetermining portion 203-3. If it is determined that the absolute valueof the difference between the pixel value of the pixel of frame #n andthe pixel value of the pixel of frame #n−1 is smaller than or equal tothe threshold Th, the stationary/moving determining portion 202-3supplies a stationary/moving determination indicating “stationary” tothe area determining portion 203-2 and the area determining portion203-3.

[0264] A stationary/moving determining portion 202-4 reads the pixelvalue of the pixel of frame #n−1 located at the same position as aspecific pixel of frame #n in which the area to which the pixel belongsis determined, and reads the pixel value of the pixel of frame #n−2located at the same position as the specific pixel of frame #n from theframe memory 201, and calculates the absolute value of the differencebetween the pixel values. The stationary/moving determining portion202-4 determines whether the absolute value of the difference betweenthe pixel value of frame #n−1 and the pixel value of frame #n−2 isgreater than a preset threshold Th. If it is determined that theabsolute value of the difference between the pixel values is greaterthan the threshold Th, a stationary/moving determination indicating“moving” is supplied to the area determining portion 203-3. If it isdetermined that the absolute value of the difference between the pixelvalue of the pixel of frame #n−1 and the pixel value of the pixel offrame #n−2 is smaller than or equal to the threshold Th, thestationary/moving determining portion 202-4 supplies a stationary/movingdetermination indicating “stationary” to the area determining portion203-3.

[0265] When the stationary/moving determination supplied from thestationary/moving determining portion 202-1 indicates “stationary” andwhen the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “moving”, the areadetermining portion 203-1 determines that the specific pixel of frame #nbelongs to an uncovered background area, and sets “1”, which indicatesthat the specific pixel belongs to an uncovered background area, in anuncovered-background-area determining flag associated with the specificpixel.

[0266] When the stationary/moving determination supplied from thestationary/moving determining portion 202-1 indicates “moving” or whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-2 indicates “stationary”, the area specifyingunit 203-1 determines that the specific pixel of frame #n does notbelong to an uncovered background area, and sets “0”, which indicatesthat the specific pixel does not belong to an uncovered background area,in the uncovered-background-area determining flag associated with thespecific pixel.

[0267] The area determining portion 203-1 supplies theuncovered-background-area determining flag in which “1” or “0” is set asdiscussed above to a determining-flag-storing frame memory 204.

[0268] When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “stationary” andwhen the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicate “stationary”, thearea determining portion 203-2 determines that the specific pixel offrame #n belongs to the stationary area, and sets “1”, which indicatesthat the pixel belongs to the stationary area, in a stationary-areadetermining flag associated with the specific pixel.

[0269] When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “moving” or whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-3 indicate “moving”, the area determiningportion 203-2 determines that the specific pixel of frame #n does notbelong to the stationary area, and sets “0”, which indicates that thepixel does not belong to the stationary area, in the stationary-areadetermining flag associated with the specific pixel.

[0270] The area determining portion 203-2 supplies the stationary-areadetermining flag in which “1” or “0” is set as discussed above to thedetermining-flag-storing frame memory 204.

[0271] When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “moving” and whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-3 indicate “moving”, the area determiningportion 203-2 determines that the specific pixel of frame #n belongs tothe moving area, and sets “1”, which indicates that the specific pixelbelongs to the moving area, in a moving-area determining flag associatedwith the specific pixel.

[0272] When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “stationary” orwhen the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicate “stationary”, thearea determining portion 203-2 determines that the specific pixel offrame #n does not belong to the moving area, and sets “0”, whichindicates that the pixel does not belong to the moving area, in themoving-area determining flag associated with the specific pixel.

[0273] The area determining portion 203-2 supplies the moving-areadetermining flag in which “1” or “0” is set as discussed above to thedetermining-flag-storing frame memory 204.

[0274] When the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicates “moving” and whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-4 indicate “stationary”, the area determiningportion 203-3 determines that the specific pixel of frame #n belongs toa covered background area, and sets “1”, which indicates that thespecific pixel belongs to the covered background area, in acovered-background-area determining flag associated with the specificpixel.

[0275] When the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicates “stationary” orwhen the stationary/moving determination supplied from thestationary/moving determining portion 202-4 indicate “moving”, the areadetermining portion 203-3 determines that the specific pixel of frame #ndoes not belong to a covered background area, and sets “0”, whichindicates that the specific pixel does not belong to a coveredbackground area, in the covered-background-area determining flagassociated with the specific pixel.

[0276] The area determining portion 203-3 supplies thecovered-background-area determining flag in which “1” or “0” is set asdiscussed above to the determining-flag-storing frame memory 204.

[0277] The determining-flag-storing frame memory 204 thus stores theuncovered-background-area determining flag supplied from the areadetermining portion 203-1, the stationary-area determining flag suppliedfrom the area determining portion 203-2, the moving-area determiningflag supplied from the area determining portion 203-2, and thecovered-background-area determining flag supplied from the areadetermining portion 203-3.

[0278] The determining-flag-storing frame memory 204 supplies theuncovered-background-area determining flag, the stationary-areadetermining flag, the moving-area determining flag, and thecovered-background-area determining flag stored therein to a synthesizer205. The synthesizer 205 generates area information indicating to whichof the uncovered background area, the stationary area, the moving area,or the covered background area each pixel belongs based on theuncovered-background-area determining flag, the stationary-areadetermining flag, the moving-area determining flag, and thecovered-background-area determining flag supplied from thedetermining-flag-storing frame memory 204, and supplies the areainformation to a determining-flag-storing frame memory 206.

[0279] The determining-flag-storing frame memory 206 stores the areainformation supplied from the synthesizer 205, and also outputs the areainformation stored therein.

[0280] An example of the processing performed by the area specifyingunit 103 is described below with reference to FIGS. 23 through 27.

[0281] When the object corresponding to the foreground is moving, theposition of the image corresponding to the object on the screen changesin every frame. As shown in FIG. 23, the image corresponding to theobject located at the position indicated by Yn(x,y) in frame #n ispositioned at Yn+1(x,y) in frame #n+1, which is subsequent to frame #n.

[0282] A model obtained by expanding in the time direction the pixelvalues of the pixels aligned side-by-side in the moving direction of theimage corresponding to the foreground object is shown in FIG. 24. Forexample, if the moving direction of the image corresponding to theforeground object is horizontal with respect to the screen, the modelshown in FIG. 24 is a model obtained by expanding in the time directionthe pixel values of the pixels disposed on a line side-by-side.

[0283] In FIG. 24, the line in frame #n is equal to the line in frame#n+1.

[0284] The foreground components corresponding to the object containedin the second pixel to the thirteenth pixel from the left in frame #nare contained in the sixth pixel through the seventeenth pixel from theleft in frame #n+1.

[0285] In frame #n, the pixels belonging to the covered background areaare the eleventh through thirteenth pixels from the left, and the pixelsbelonging to the uncovered background area are the second through fourthpixels from the left. In frame #n+1, the pixels belonging to the coveredbackground area are the fifteenth through seventeenth pixels from theleft, and the pixels belonging to the uncovered background area are thesixth through eighth pixels from the left.

[0286] In the example shown in FIG. 24, since the foreground componentscontained in frame #n are moved by four pixels in frame #n+1, the amountof movement v is 4. The number of virtual divided portions is 4 inaccordance with the amount of movement v.

[0287] A description is now given of a change in pixel values of thepixels belonging to the mixed area in the frames before and after aspecific frame.

[0288] In FIG. 25, the pixels belonging to a covered background area inframe #n in which the background is stationary and the amount ofmovement v in the foreground is 4 are the fifteenth through seventeenthpixels from the left. Since the amount of movement v is 4, the fifteenththrough seventeenth frames from the left in the previous frame #n−1contain only background components and belong to the background area.The fifteenth through seventeenth pixels from the left in frame #n−2,which is one before frame #n−1, contain only background components andbelong to the background area.

[0289] Since the object corresponding to the background is stationary,the pixel value of the fifteenth pixel from the left in frame #n−1 doesnot change from the pixel value of the fifteenth pixel from the left inframe #n−2. Similarly, the pixel value of the sixteenth pixel from theleft in frame #n−1 does not change from the pixel value of the sixteenthpixel from the left in frame #n−2, and the pixel value of theseventeenth pixel from the left in frame #n−1 does not change from thepixel value of the seventeenth pixel from the left in frame #n−2.

[0290] That is, the pixels in frame #n−1 and frame #n−2 corresponding tothe pixels belonging to the covered background area in frame #n consistof only background components, and the pixel values thereof do notchange. Accordingly, the absolute value of the difference between thepixel values is almost 0. Thus, the stationary/moving determination madefor the pixels in frame #n−1 and frame #n−2 corresponding to the pixelsbelonging to the mixed area in frame #n by the stationary/movingdetermining portion 202-4 is “stationary”.

[0291] Since the pixels belonging to the covered background area inframe #n contain foreground components, the pixel values thereof aredifferent from those of frame #n−1 consisting of only backgroundcomponents. Accordingly, the stationary/moving determination made forthe pixels belonging to the mixed area in frame #n and the correspondingpixels in frame #n−1 by the stationary/moving determining portion 202-3is “moving”.

[0292] When the stationary/moving determination result indicating“moving” is supplied from the stationary/moving determining portion202-3, and when the stationary/moving determination result indicating“stationary” is supplied from the stationary/moving determining portion202-4, as discussed above, the area determining portion 203-3 determinesthat the corresponding pixels belong to a covered background area.

[0293] In FIG. 26, in frame #n in which the background is stationary andthe amount of movement v in the foreground is 4, the pixels contained inan uncovered background area are the second through fourth pixels fromthe left. Since the amount of movement v is 4, the second through fourthpixels from the left in the subsequent frame #n+1 contain onlybackground components and belong to the background area. In frame #n+2,which is subsequent to frame #n+1, the second through fourth pixels fromthe left contain only background components and belong to the backgroundarea.

[0294] Since the object corresponding to the background is stationary,the pixel value of the second pixel from the left in frame #n+2 does notchange from the pixel value of the second pixel from the left in frame#n+1. Similarly, the pixel value of the third pixel from the left inframe #n+2 does not change from the pixel value of the third pixel fromthe left in frame #n+1, and the pixel value of the fourth pixel from theleft in frame #n+2 does not change from the pixel value of the fourthpixel from the left in frame #n+1.

[0295] That is, the pixels in frame #n+1 and frame #n+2 corresponding tothe pixels belonging to the uncovered background area in frame #nconsist of only background components, and the pixel values thereof donot change. Accordingly, the absolute value of the difference betweenthe pixel values is almost 0. Thus, the stationary/moving determinationmade for the pixels in frame #n+1 and frame #n+2 corresponding to thepixels belonging to the mixed area in frame #n by the stationary/movingdetermining portion 202-1 is “stationary”.

[0296] Since the pixels belonging to the uncovered background area inframe #n contain foreground components, the pixel values thereof aredifferent from those of frame #n+1 consisting of only backgroundcomponents. Accordingly, the stationary/moving determination made forthe pixels belonging to the mixed area in frame #n and the correspondingpixels in frame #n+1 by the stationary/moving determining portion 202-2is “moving”.

[0297] When the stationary/moving determination result indicating“moving” is supplied from the stationary/moving determining portion202-2, and when the stationary/moving determination result indicating“stationary” is supplied from the stationary/moving determining portion202-1, as discussed above, the area determining portion 203-1 determinesthat the corresponding pixels belong to an uncovered background area.

[0298]FIG. 27 illustrates determination conditions for frame #n made bythe area specifying unit 103. When the determination result for thepixel in frame #n−2 located at the same image position as a pixel inframe #n to be processed and for the pixel in frame #n−1 located at thesame position as the pixel in frame #n is stationary, and when thedetermination result for the pixel in frame #n and the pixel in frame#n−1 located at the same image position as the pixel in frame #n ismoving, the area specifying unit 103 determines that the pixel in frame#n belongs to a covered background area.

[0299] When the determination result for the pixel in frame #n and thepixel in frame #n−1 located at the same image position as the pixel inframe #n is stationary, and when the determination result for the pixelin frame #n and the pixel in frame #n+1 located at the same imageposition as the pixel in frame #n is stationary, the area specifyingunit 103 determines that the pixel in frame #n belongs to the stationaryarea.

[0300] When the determination result for the pixel in frame #n and thepixel in frame #n−1 located at the same image position as the pixel inframe #n is moving, and when the determination result for the pixel inframe #n and the pixel in frame #n+1 located at the same image positionas the pixel in frame #n is moving, the area specifying unit 103determines that the pixel in frame #n belongs to the moving area.

[0301] When the determination result for the pixel in frame #n and thepixel in frame #n+1 located at the same image position as the pixel inframe #n is moving, and when the determination result for the pixel inframe #n+1 located at the same image position as the pixel in frame #nand the pixel in frame #n+2 located at the same image position as thepixel in frame #n is stationary, the area specifying unit 103 determinesthat the pixel in frame #n belongs to an uncovered background area.

[0302]FIGS. 28A through 28D illustrate examples of the areadetermination results obtained by the area specifying unit 103. In FIG.28A, the pixels which are determined to belong to a covered backgroundarea are indicated in white. In FIG. 28B, the pixels which aredetermined to belong to an uncovered background area are indicated inwhite.

[0303] In FIG. 28C, the pixels which are determined to belong to amoving area are indicated in white. In FIG. 28D, the pixels which aredetermined to belong to a stationary area are indicated in white.

[0304]FIG. 29 illustrates the area information indicating the mixedarea, in the form of an image, selected from the area information outputfrom the determining-flag-storing frame memory 206. In FIG. 29, thepixels which are determined to belong to the covered background area orthe uncovered background area, i.e., the pixels which are determined tobelong to the mixed area, are indicated in white. The area informationindicating the mixed area output from the determining-flag-storing framememory 206 designates the mixed area and the portions having a texturesurrounded by the portions without a texture in the foreground area.

[0305] The area specifying processing performed by the area specifyingunit 103 is described below with reference to the flowchart of FIG. 30.In step S201, the frame memory 201 obtains an image of frame #n−2through frame #n+2 including frame #n.

[0306] In step S202, the stationary/moving determining portion 202-3determines whether the determination result for the pixel in frame #n−1and the pixel in frame #n located at the same position is stationary. Ifit is determined that the determination result is stationary, theprocess proceeds to step S203 in which the stationary/moving determiningportion 202-2 determines whether the determination result for the pixelin frame #n and the pixel in frame #n+1 located at the same position isstationary.

[0307] If it is determined in step S203 that the determination resultfor the pixel in frame #n and the pixel in frame #n+1 located at thesame position is stationary, the process proceeds to step S204. In stepS204, the area determining portion 203-2 sets “1”, which indicates thatthe pixel to be processed belongs to the stationary area, in thestationary-area determining flag associated with the pixel to beprocessed. The area determining portion 203-2 supplies thestationary-area determining flag to the determining-flag-storing framememory 204, and the process proceeds to step S205.

[0308] If it is determined in step S202 that the determination resultfor the pixel in frame #n−1 and the pixel in frame #n located at thesame position is moving, or if it is determined in step S203 that thedetermination result for the pixel in frame #n and the pixel in frame#n+1 located at the same position is moving, the pixel to be processeddoes not belong to a stationary area. Accordingly, the processing ofstep S204 is skipped, and the process proceeds to step S205.

[0309] In step S205, the stationary/moving determining portion 202-3determines whether the determination result for the pixel in frame #n−1and the pixel in frame #n located at the same position is moving. If itis determined that the determination result is moving, the processproceeds to step S206 in which the stationary/moving determining portion202-2 determines whether the determination result for the pixel in frame#n and the pixel in frame #n+1 located at the same position is moving.

[0310] If it is determined in step S206 that the determination resultfor the pixel in frame #n and the pixel in frame #n+1 located at thesame position is moving, the process proceeds to step S207. In stepS207, the area determining portion 203-2 sets “1”, which indicates thatthe pixel to be processed belongs to a moving area, in the moving-areadetermining flag associated with the pixel to be processed. The areadetermining area 203-2 supplies the moving-area determining flag to thedetermining-flag-storing frame memory 204, and the process proceeds tostep S208.

[0311] If it is determined in step S205 that the determination resultfor the pixel in frame #n−1 and the pixel in frame #n located at thesame position is stationary, or if it is determined in step S206 thatthe determination result for the pixel in frame #n and the pixel inframe #n+1 located at the same position is stationary, the pixel inframe #n does not belong to a moving area. Accordingly, the processingof step S207 is skipped, and the process proceeds to step S208.

[0312] In step S208, the stationary/moving determining portion 202-4determines whether the determination result for the pixel in frame #n−2and the pixel in frame #n−1 located at the same position is stationary.If it is determined that the determination result is stationary, theprocess proceeds to step S209 in which the stationary/moving determiningportion 202-3 determines whether the determination result for the pixelin frame #n−1 and the pixel in frame #n located at the same position ismoving.

[0313] If it is determined in step S209 that the determination resultfor the pixel in frame #n−1 and the pixel in frame #n located at thesame position is moving, the process proceeds to step S210. In stepS210, the area determining,portion 203-3 sets “1”, which indicates thatthe pixel to be processed belongs to a covered background area, in thecovered-background-area determining flag associated with the pixel to beprocessed. The area determining portion 203-3 supplies thecovered-background-area determining flag to the determining-flag-storingframe memory 204, and the process proceeds to step S211. The areadetermining portion 203-3 supplies the covered-background-areadetermining flag to the determining-flag-storing frame memory 204, andthe process proceeds to step S211.

[0314] If it is determined in step S208 that the determination resultfor the pixel in frame #n−2 and the pixel in frame #n−1 located at thesame position is moving, or if it is determined in step S209 that thepixel in frame #n−1 and the pixel in frame #n located at the sameposition is stationary, the pixel in frame #n does not belong to acovered background area. Accordingly, the processing of step S210 isskipped, and the process proceeds to step S211.

[0315] In step S211, the stationary/moving determining portion 202-2determines whether the determination result for the pixel in frame #nand the pixel in frame #n+1 located at the same position is moving. Ifit is determined in step S211 that the determination result is moving,the process proceeds to step S212 in which the stationary/movingdetermining portion 202-1 determines whether the determination resultfor the pixel in frame #n+1 and the pixel in frame #n+2 located at thesame position is stationary.

[0316] If it is determined in step S212 that the determination resultfor the pixel in frame #n+1 and the pixel in frame #n+2 located at thesame position is stationary, the process proceeds to step S213. In stepS213, the area determining portion 203-1 sets “1”, which indicates thatthe pixel to be processed belongs to an uncovered background area, inthe uncovered-background-area determining flag associated with the pixelto be processed. The area determining portion 203-1 supplies theuncovered-background-flag determining flag to thedetermining-flag-storing frame memory 204, and the process proceeds tostep S214.

[0317] If it is determined in step S211 that the determination resultfor the pixel in frame #n and the pixel in frame #n+1 located at thesame position is stationary, or if it is determined in step S212 thatthe determination result for the pixel in frame #n+1 and the pixel inframe #n+2 is moving, the pixel in frame #n does not belong to anuncovered background area. Accordingly, the processing of step S213 isskipped, and the process proceeds to step S214.

[0318] In step S214, the area specifying unit 103 determines whether theareas of all the pixels in frame #n are specified. If it is determinedthat the areas of all the pixels in frame #n are not yet specified, theprocess returns to step S202, and the area specifying processing isrepeated for the remaining pixels.

[0319] If it is determined in step S214 that the areas of all the pixelsin frame #n are specified, the process proceeds to step S215. In stepS215, the synthesizer 215 generates area information indicating themixed area based on the uncovered-background-area determining flag andthe covered-background-area determining flag stored in thedetermining-flag-storing frame memory 204, and also generates areainformation indicating to which of the uncovered background area, thestationary area, the moving area, or the covered background area eachpixel belongs, and sets the generated area information in thedetermining-flag-storing frame memory 206. The processing is thencompleted.

[0320] As discussed above, the area specifying unit 103 is capable ofgenerating area information indicating to which of the moving area, thestationary area, the uncovered background area, or the coveredbackground area each of the pixels contained in a frame belongs.

[0321] The area specifying unit 103 may apply logical OR to the areainformation corresponding to the uncovered background area and the areainformation corresponding to the covered background area so as togenerate area information corresponding to the mixed area, and then maygenerate area information consisting of flags indicating to which of themoving area, the stationary area, or the mixed area the individualpixels contained in the frame belong.

[0322] When the object corresponding to the foreground has a texture,the area specifying unit 103 is able to specify the moving area moreprecisely.

[0323] The area specifying unit 103 is able to output the areainformation indicating the moving area as the area informationindicating the foreground area, and outputs the area informationindicating the stationary area as the area information indicating thebackground area.

[0324] The embodiment has been described, assuming that the objectcorresponding to the background is stationary. However, theabove-described area specifying processing can be applied even if theimage corresponding to the background area contains motion. For example,if the image corresponding to the background area is uniformly moving,the area specifying unit 103 shifts the overall image in accordance withthis motion, and performs processing in a manner similar to the case inwhich the object corresponding to the background is stationary. If theimage corresponding to the background area contains locally differentmotions, the area specifying unit 103 selects the pixels correspondingto the motions, and executes the above-described processing.

[0325]FIG. 31 is a block diagram illustrating an example of anotherconfiguration of the area specifying unit 103. The area specifying unit103 shown in FIG. 31 does not use a motion vector. A background imagegenerator 301 generates a background image corresponding to an inputimage, and supplies the generated background image to abinary-object-image extracting portion 302. The background imagegenerator 301 extracts, for example, an image object corresponding to abackground object contained in the input image, and generates thebackground image.

[0326] An example of a model obtained by expanding in the time directionthe pixel values of pixels aligned side-by-side in the moving directionof an image corresponding to a foreground object is shown in FIG. 32.For example, if the moving direction of the image corresponding to theforeground object is horizontal with respect to the screen, the modelshown in FIG. 32 is a model obtained by expanding the pixel values ofpixels disposed side-by-side on a single line in the time domain.

[0327] In FIG. 32, the line in frame #n is the same as the line in frame#n−1 and the line in frame #n+1.

[0328] In frame #n, the foreground components corresponding to theobject contained in the sixth through seventeenth pixels from the leftare contained in the second through thirteenth pixels from the left inframe #n−1 and are also contained in the tenth through twenty-firstpixel from the left in frame #n+1.

[0329] In frame #n−1, the pixels belonging to the covered backgroundarea are the eleventh through thirteenth pixels from the left, and thepixels belonging to the uncovered background area are the second throughfourth pixels from the left. In frame #n, the pixels belonging to thecovered background area are the fifteenth through seventeenth pixelsfrom the left, and the pixels belonging to the uncovered background areaare the sixth through eighth pixels from the left. In frame #n+1, thepixels belonging to the covered background area are the nineteenththrough twenty-first pixels from the left, and the pixels belonging tothe uncovered background area are the tenth through twelfth pixels fromthe left.

[0330] In frame #n−1, the pixels belonging to the background area arethe first pixel from the left, and the fourteenth through twenty-firstpixels from the left. In frame #n, the pixels belonging to thebackground area are the first through fifth pixels from the left, andthe eighteenth through twenty-first pixels from the left. In frame #n+1,the pixels belonging to the background area are the first through ninthpixels from the left.

[0331] An example of the background image corresponding to the exampleshown in FIG. 32 generated by the background image generator 301 isshown in FIG. 33. The background image consists of the pixelscorresponding to the background object, and does not contain imagecomponents corresponding to the foreground object.

[0332] The binary-object-image extracting portion 302 generates a binaryobject image based on the correlation between the background image andthe input image, and supplies the generated binary object image to atime change detector 303.

[0333]FIG. 34 is a block diagram illustrating the configuration of thebinary-object-image extracting portion 302. A correlation-valuecalculator 321 calculates the correlation between the background imagesupplied from the background image generator 301 and the input image soas to generate a correlation value, and supplies the generatedcorrelation value to a threshold-value processor 322.

[0334] The correlation-value calculator 321 applies equation (4) to, forexample, 3×3-background image blocks having X₄ at the center, as shownin FIG. 35A, and to, for example, 3×3-background image blocks having Y₄at the center which corresponds to the background image blocks, as shownin FIG. 35B, thereby calculating a correlation value corresponding toY₄. $\begin{matrix}{{{Correlation}\quad {value}} = \frac{\sum\limits_{i = 0}^{8}{( {{Xi} - \overset{\_}{X}} ){\sum\limits_{i = 0}^{8}( {{Yi} - \overset{\_}{Y}} )}}}{\sqrt{\sum\limits_{i = 0}^{8}{( {{Xi} - \overset{\_}{X}} )^{2} \cdot {\sum\limits_{i = 0}^{8}( {{Yi} - \overset{\_}{Y}} )^{2}}}}}} & (4) \\{\overset{\_}{X} = \frac{\sum\limits_{i = 0}^{8}{Xi}}{9}} & (5) \\{\overset{\_}{Y} = \frac{\sum\limits_{i = 0}^{8}{Yi}}{9}} & (6)\end{matrix}$

[0335] The correlation-value calculator 321 supplies the correlationvalue calculated for each pixel as discussed above to thethreshold-value processor 322.

[0336] Alternatively, the correlation-value calculator 321 may applyequation (7) to, for example, 3×3-background image blocks having X₄ atthe center, as shown in FIG. 36A, and to, for example, 3×3-backgroundimage blocks having Y₄ at the center which corresponds to the backgroundimage blocks, as shown in FIG. 36B, thereby calculating the sum ofabsolute values of differences corresponding to Y₄. $\begin{matrix}{{{Sum}\quad {of}\quad {absolute}\quad {values}\quad {of}\quad {differences}} = {\sum\limits_{i = 0}^{8}| ( {{Xi} - {Yi}} ) |}} & (7)\end{matrix}$

[0337] The correlation-value calculator 321 supplies the sum of theabsolute values of the differences calculated as described above to thethreshold-value processor 322 as the correlation value.

[0338] The threshold-value processor 322 compares the pixel value of thecorrelation image with a threshold value th0. If the correlation valueis smaller than or equal to the threshold value th0, 1 is set in thepixel value of the binary object image. If the correlation value isgreater than the threshold value th0, 0 is set in the pixel value of thebinary object image. The threshold-value processor 322 then outputs thebinary object image whose pixel value is set to 0 or 1. Thethreshold-value processor 322 may store the threshold value th0 thereinin advance, or may use the threshold value th0 input from an externalsource.

[0339]FIG. 37 illustrates the binary object image corresponding to themodel of the input image shown in FIG. 32. In the binary object image, 0is set in the pixel values of the pixels having a higher correlationwith the background image.

[0340]FIG. 38 is a block diagram illustrating the configuration of thetime change detector 303. When determining the area of a pixel in frame#n, a frame memory 341 stores a binary object image of frame #n−1, frame#n, and frame #n+1 supplied from the binary-object-image extractingportion 302.

[0341] An area determining portion 342 determines the area of each pixelof frame #n based on the binary object image of frame #n−1, frame #n,and frame #n+1 so as to generate area information, and outputs thegenerated area information.

[0342]FIG. 39 illustrates the determinations made by the areadetermining portion 342. When the pixel of interest of the binary objectimage in frame #n is 0, the area determining portion 342 determines thatthe pixel of interest in frame #n belongs to the background area.

[0343] When the pixel of interest of the binary object image in frame #nis 1, and when the corresponding pixel of the binary object image inframe #n−1 is 1, and when the corresponding pixel of the binary objectimage in frame #n+1 is 1, the area determining portion 342 determinesthat the pixel of interest in frame #n belongs to the foreground area.

[0344] When the pixel of interest of the binary object image in frame #nis 1, and when the corresponding pixel of the binary object image inframe #n−1 is 0, the area determining portion 342 determines that thepixel of interest in frame #n belongs to a covered background area.

[0345] When the pixel of interest of the binary object image in frame #nis 1, and when the corresponding pixel of the binary object image inframe #n+1 is 0, the area determining portion 342 determines that thepixel of interest in frame #n belongs to an uncovered background area.

[0346]FIG. 40 illustrates an example of the determinations made by thetime change detector 303 on the binary object image corresponding to themodel of the input image shown in FIG. 32. The time change detector 303determines that the first through fifth pixels from the left in frame #nbelong to the background area since the corresponding pixels of thebinary object image in frame #n are 0.

[0347] The time change detector 303 determines that the sixth throughninth pixels from the left belong to the uncovered background area sincethe pixels of the binary object image in frame #n are 1, and thecorresponding pixels in frame #n+1 are 0.

[0348] The time change detector 303 determines that the tenth throughthirteenth pixels from the left belong to the foreground area since thepixels of the binary object image in frame #n are 1, the correspondingpixels in frame #n−1 are 1, and the corresponding pixels in frame #n+1are 1.

[0349] The time change detector 303 determines that the fourteenththrough seventeenth pixels from the left belong to the coveredbackground area since the pixels of the binary object image in frame #nare 1, and the corresponding pixels in frame #n−1 are 0.

[0350] The time change detector 303 determines that the eighteenththrough twenty-first pixels from the-left belong to the background areasince the corresponding pixels of the binary object image in frame #nare 0.

[0351] The area specifying processing performed by the area specifyingunit 103 is described below with reference to the flowchart of FIG. 41.In step S301, the background image generator 301 of the area specifyingunit 103 extracts, for example, an image object corresponding to abackground object contained in an input image based on the input imageso as to generate a background image, and supplies the generatedbackground image to the binary-object-image extracting portion 302.

[0352] In step S302, the binary-object-image extracting portion 302calculates a correlation value between the input image and thebackground image supplied from the background image generator 301according to, for example, calculation discussed with reference to FIGS.35A and 35B. In step S303, the binary-object-image extracting portion302 computes a binary object image from the correlation value and thethreshold value th0 by, for example, comparing the correlation valuewith the threshold value th0.

[0353] In step S304, the time change detector 303 executes the areadetermining processing, and the processing is completed.

[0354] Details of the area determining processing in step S304 aredescribed below with reference to the flowchart of FIG. 42. In stepS321, the area determining portion 342 of the time change detector 303determines whether the pixel of interest in frame #n stored in the framememory 341 is 0. If it is determined that the pixel of interest in frame#n is 0, the process proceeds to step S322. In step S322, it isdetermined that the pixel of interest in frame #n belongs to thebackground area, and the processing is completed.

[0355] If it is determined in step S321 that the pixel of interest inframe #n is 1, the process proceeds to step S323. In step S323, the areadetermining portion 342 of the time change detector 303 determineswhether the pixel of interest in frame #n stored in the frame memory 341is 1, and whether the corresponding pixel in frame #n−1 is 0. If it isdetermined that the pixel of interest in frame #n is 1 and thecorresponding pixel in frame #n−1 is 0, the process proceeds to stepS324. In step S324, it is determined that the pixel of interest in frame#n belongs to the covered background area, and the processing iscompleted.

[0356] If it is determined in step S323 that the pixel of interest inframe #n is 0, or that the corresponding pixel in frame #n−1 is 1, theprocess proceeds to step S325. In step S325, the area determiningportion 342 of the time change detector 303 determines whether the pixelof interest in frame #n stored in the frame memory 341 is 1, and whetherthe corresponding pixel in frame #n+1 is 0. If it is determined that thepixel of interest in frame #n is 1 and the corresponding pixel in frame#n+1 is 0, the process proceeds to step S326. In step S326, it isdetermined that the pixel of interest in frame #n belongs to theuncovered background area, and the processing is completed.

[0357] If it is determined in step S325 that the pixel of interest inframe #n is 0, or that the corresponding pixel in frame #n+1 is 1, theprocess proceeds to step S327. In step S327, the area determiningportion 342 of the time change detector 303 determines that the pixel ofinterest in frame #n belongs to the foreground area, and the processingis completed.

[0358] As discussed above, the area specifying unit 103 is able tospecify, based on the correlation value between the input image and thecorresponding background image, to which of the foreground area, thebackground area, the covered background area, or the uncoveredbackground area each pixel of the input image belongs, and generatesarea information corresponding to the specified result.

[0359]FIG. 43 is a block diagram illustrating another configuration ofthe area specifying unit 103. The area specifying unit 103 uses a motionvector and positional information thereof supplied from the motiondetector 102. The same elements as those shown in FIG. 31 are designatedwith like reference numerals, and an explanation thereof is thusomitted.

[0360] A robust-processing portion 361 generates a robust binary objectimage based on binary object images of N frames supplied from thebinary-object-image extracting portion 302, and outputs the robustbinary object image to the time change detector 303.

[0361]FIG. 44 is a block diagram illustrating the configuration of therobust-processing portion 361. A motion compensator 381 compensates forthe motion of the binary object images of N frames based on the motionvector and the positional information thereof supplied from the motiondetector 102, and outputs a motion-compensated binary object image to aswitch 382.

[0362] The motion compensation performed by the motion compensator 381is discussed below with reference to examples shown in FIGS. 45 and 46.It is now assumed, for example, that the area in frame #n is to beprocessed. When binary object images of frame #n−1, frame #n, and frame#n+1 shown in FIG. 45 are input, the motion compensator 381 compensatesfor the motion of the binary object image of frame #n−1 and the binaryobject image of frame #n+1, as indicated by the example shown in FIG.46, based on the motion vector supplied from the motion detector 102,and supplies the motion-compensated binary object images to the switch382.

[0363] The switch 382 outputs the motion-compensated binary object imageof the first frame to a frame memory 383-1, and outputs themotion-compensated binary object image of the second frame to a framememory 383-2. Similarly, the switch 382 outputs the motion-compensatedbinary object images of the third through (N−1)-th frame to framememories 383-3 through 383-(N−1), and outputs the motion-compensatedbinary object image of the N-th frame to a frame memory 383-N.

[0364] The frame memory 383-1 stores the motion-compensated binaryobject image of the first frame, and outputs the stored binary objectimage to a weighting portion 384-1. The frame memory 383-2 stores themotion-compensated binary object image of the second frame, and outputsthe stored binary object image to a weighting portion 384-2.

[0365] Similarly, the frame memories 383-3 through 383-(N−1) stores themotion-compensated binary object images of the third through (N−1)-thframes, and outputs the stored binary object images to weightingportions 384-3 through 384-(N−1). The frame memory 383-N stores themotion-compensated binary object image of the N-th frame, and outputsthe stored binary object image to a weighting portion 384-N.

[0366] The weighting portion 384-1 multiplies the pixel value of themotion-compensated binary object image of the first frame supplied fromthe frame memory 383-1 by a predetermined weight w1, and supplies aweighted binary object image to an accumulator 385. The weightingportion 384-2 multiplies the pixel value of the motion-compensatedbinary object image of the second frame supplied from the frame memory383-2 by a predetermined weight w2, and supplies the weighted binaryobject image to the accumulator 385.

[0367] Likewise, the weighting portions 384-3 through 384-(N−1) multiplythe pixel values of the motion-compensated binary object images of thethird through (N−1)-th frames supplied from the frame memories 383-3through 383-(N−1) by predetermined weights w3 through w(N−1), andsupplies the weighted binary object images to the accumulator 385. Theweighting portion 384-N multiplies the pixel value of themotion-compensated binary object image of the N-th frame supplied fromthe frame memory 383-N by a predetermined weight wN, and supplies theweighted binary object image to the accumulator 385.

[0368] The accumulator 385 accumulates the pixel values of themotion-compensated binary object images multiplied by the weights w1through wN of the first through N-th frames, and compares theaccumulated pixel value with the predetermined threshold value th0,thereby generating the binary object image.

[0369] As discussed above, the robust-processing portion 361 generates arobust binary object image from N binary object images, and supplies itto the time change detector 303. Accordingly, the area specifying unit103 configured as shown in FIG. 43 is able to specify the area moreprecisely than that shown in FIG. 31 even if noise is contained in theinput image.

[0370] The area specifying processing performed by the area specifyingunit 103 configured as shown in FIG. 43 is described below withreference to the flowchart of FIG. 47. The processings of step S341through step S343 are similar to those of step S301 through step S303discussed with reference to the flowchart of FIG. 41, and an explanationthereof is thus omitted.

[0371] In step S344, the robust-processing portion 361 performs therobust processing.

[0372] In step S345, the time change detector 303 performs the areadetermining processing, and the processing is completed. Details of theprocessing of step S345 are similar to the processing discussed withreference to the flowchart of FIG. 42, and an explanation thereof isthus omitted.

[0373] Details of the robust processing corresponding to the processingof step S344 in FIG. 47 are given below with reference to the flowchartof FIG. 48. In step S361, the motion compensator 381 performs the motioncompensation of an input binary object image based on the motion vectorand the positional information thereof supplied from the motion detector102. In step S362, one of the frame memories 383-1 through 383-N storesthe corresponding motion-compensated binary object image supplied viathe switch 382.

[0374] In step S363, the robust-processing portion 361 determineswhether N binary object images are stored. If it is determined that Nbinary object images are not stored, the process returns to step S361,and the processing for compensating for the motion of the binary objectimage and the processing for storing the binary object image arerepeated.

[0375] If it is determined in step S363 that N binary object images arestored, the process proceeds to step S364 in which weighting isperformed. In step S364, the weighting portions 384-1 through 384-Nmultiply the corresponding N binary object images by the weights w1through wN.

[0376] In step S365, the accumulator 385 accumulates the N weightedbinary object images.

[0377] In step S366, the accumulator 385 generates a binary object imagefrom the accumulated images by, for example, comparing the accumulatedvalue with a predetermined threshold value th1, and the processing iscompleted.

[0378] As discussed above, the area specifying unit 103 configured asshown in FIG. 43 is able to generate area information based on therobust binary object image.

[0379] As is seen from the foregoing description, the area specifyingunit 103 is able to generate area information indicating to which of themoving area, the stationary area, the uncovered background area, or thecovered background area each pixel contained in a frame belongs.

[0380]FIG. 49 is a block diagram illustrating the configuration of themixture-ratio calculator 104. An estimated-mixture-ratio processor 401calculates an estimated mixture ratio for each pixel by calculating amodel of a covered background area based on the input image, andsupplies the calculated estimated mixture ratio to a mixture-ratiodetermining portion 403.

[0381] An estimated-mixture-ratio processor 402 calculates an estimatedmixture ratio for each pixel by calculating a model of an uncoveredbackground area based on the input image, and supplies the calculatedestimated mixture ratio to the mixture-ratio determining portion 403.

[0382] Since it can be assumed that the object corresponding to theforeground is moving with constant velocity within the shutter time, themixture ratio α of the pixels belonging to a mixed area exhibits thefollowing characteristics. That is, the mixture ratio α linearly changesaccording to the positional change in the pixels. If the positionalchange in the pixels is one-dimensional, a change in the mixture ratio αcan be represented linearly. If the positional change in the pixels istwo-dimensional, a change in the mixture ratio α can be represented on aplane.

[0383] Since the period of one frame is short, it can be assumed thatthe object corresponding to the foreground is a rigid body moving withconstant velocity.

[0384] The gradient of the mixture ratio α is inversely proportional tothe amount of movement v within the shutter time of the foreground.

[0385] An example of the ideal mixture ratio α is shown in FIG. 50. Thegradient 1 of the ideal mixture ratio α in the mixed area can berepresented by the reciprocal of the amount of movement v.

[0386] As shown in FIG. 50, the ideal mixture ratio α has the value of 1in the background area, the value of 0 in the foreground area, and thevalue of greater than 0 and smaller than 1 in the mixed area.

[0387] In the example shown in FIG. 51, the pixel value C06 of theseventh pixel from the left in frame #n can be indicated by equation (8)by using the pixel value P06 of the seventh pixel from the left in frame#n−1. $\begin{matrix}\begin{matrix}{{C06} = {{{B06}/v} + {{B06}/v} + {{F01}/v} + {{F02}/v}}} \\{= {{{P06}/v} + {{P06}/v} + {{F01}/v} + {{F02}/v}}} \\{= {{{2/v} \cdot {P06}} + {\sum\limits_{i = 1}^{2}{{Fi}/v}}}}\end{matrix} & (8)\end{matrix}$

[0388] In equation (8), the pixel value C06 is indicated by a pixelvalue M of the pixel in the mixed area, while the pixel value P06 isindicated by a pixel value B of the pixel in the background area. Thatis, the pixel value M of the pixel in the mixed area and the pixel valueB of the pixel in the background area can be represented by equations(9) and (10), respectively.

M=C06  (9)

B=P06  (10)

[0389] In equation (8), 2/v corresponds to the mixture ratio α. Sincethe amount of movement v is 4, the mixture ratio α of the seventh pixelfrom the left in frame #n is 0.5.

[0390] As discussed above, the pixel value C in frame #n of interest isconsidered as the pixel value in the mixed area, while the pixel value Pof frame #n−1 prior to frame #n is considered as the pixel value in thebackground area. Accordingly, equation (3) indicating the mixture ratioα can be represented by equation (11):

C=α·P+f  (11)

[0391] where f in equation (11) indicates the sum of the foregroundcomponents Σ_(i)Fi/v contained in the pixels of interest. The variablescontained in equation (11) are two factors, i.e., the mixture ratio αand the sum f of the foreground components.

[0392] Similarly, a model obtained by expanding in the time directionthe pixel values in which the amount of movement is 4 and the number ofvirtual divided portions is 4 in an uncovered background area is shownin FIG. 52.

[0393] As in the representation of the covered background area, in theuncovered background area, the pixel value C of frame #n of interest isconsidered as the pixel value in the mixed area, while the pixel value Nof frame #n+1 subsequent to frame #n is considered as the backgroundarea. Accordingly, equation (3) indicating the mixture ratio α can berepresented by equation (12).

C=α·N+f  (12)

[0394] The embodiment has been described, assuming that the backgroundobject is stationary. However, equations (8) through (12) can be appliedto the case in which the background object is moving by using the pixelvalue of a pixel located corresponding to the amount of movement v ofthe background. It is now assumed, for example, in FIG. 51 that theamount of movement v of the object corresponding to the background is 2,and the number of virtual divided portions is 2. In this case, when theobject corresponding to the background is moving to the right in FIG.49, the pixel value B of the pixel in the background area in equation(10) is represented by a pixel value P04.

[0395] Since equations (11) and (12) each contain two variables, themixture ratio α cannot be determined without modifying the equations.Generally, an image has a strong spatial correlation, and accordingly,pixels located in close proximity with each other have almost the samepixel values.

[0396] Since the foreground components have a spatially strongcorrelation, the equation is modified so that the sum f of theforeground components can be deduced from the previous or subsequentframe, thereby determining the mixture ratio α.

[0397] The pixel value Mc of the seventh pixel from the left in frame #nin FIG. 53 can be expressed by equation (13). $\begin{matrix}{{Mc} = {{\frac{2}{v} \cdot {B06}} + {\sum\limits_{i = 11}^{12}{{Fi}/v}}}} & (13)\end{matrix}$

[0398] The first term 2/v of the right side in equation (13) correspondsto the mixture ratio α. The second term of the right side in equation(13) can be expressed by equation (14) by utilizing the pixel value inthe subsequent frame $\begin{matrix}{{{\# n} + 1.}{{\sum\limits_{i = 11}^{12}{{Fi}/v}} = {\beta \cdot {\sum\limits_{i = 7}^{10}{{Fi}/v}}}}} & (14)\end{matrix}$

[0399] It is now assumed that equation (15) holds true by utilizing thespatial correlation of the foreground component.

F=F05=F06=F07=F08=F09=F10=F11=F12  (15)

[0400] Equation (14) can be modified into equation (16) by utilizingequation (15). $\begin{matrix}\begin{matrix}{{\sum\limits_{i = 11}^{12}{{Fi}/v}} = {\frac{2}{v} \cdot F}} \\{= {\beta \cdot \frac{4}{v} \cdot F}}\end{matrix} & (16)\end{matrix}$

[0401] As a result, β can be expressed by equation (17).

β=2/4  (17)

[0402] If it is assumed that the foreground components in the mixed areaare equal, as indicated by equation (15), equation (18) can hold truefor all the pixels in the mixed area because of the internal ratio.

β=1−α  (18)

[0403] If equation (18) holds true, equation (11) can be developed intoequation (19). $\begin{matrix}\begin{matrix}{C = {{\alpha \cdot P} + f}} \\{= {{\alpha \cdot P} + {( {1 - \alpha} ) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}{{Fi}/v}}}}} \\{= {{\alpha \cdot P} + {( {1 - \alpha} ) \cdot N}}}\end{matrix} & (19)\end{matrix}$

[0404] Similarly, if equation (18) holds true, equation (12) can bedeveloped into equation (20). $\begin{matrix}\begin{matrix}{C = {{\alpha \cdot N} + f}} \\{= {{\alpha \cdot N} + {( {1 - \alpha} ) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}{{Fi}/v}}}}} \\{= {{\alpha \cdot N} + {( {1 - \alpha} ) \cdot P}}}\end{matrix} & (20)\end{matrix}$

[0405] In equations (19) and (20), since C, N, and P are known pixelvalues, the variable contained in equations (19) and (20) is only themixture ratio α. The relationship among C, N, and P in equations (19)and (20) is shown in FIG. 54. C is the pixel value of the pixel ofinterest in frame #n for which the mixture ratio α is calculated. N isthe pixel value of the pixel in frame #n+1 located at the positionspatially corresponding to the pixel of interest. P is the pixel valueof the pixel in frame #n−1 located at the position spatiallycorresponding to the pixel of interest.

[0406] Accordingly, since one variable is contained in each of equations(19) and (20), the mixture ratio α is calculated by utilizing the pixelsin the three frames. The condition for solving the correct mixture ratioα by solving equations (19) and (20) is as follows. In the image objecthaving the same foreground components in the mixed area, i.e., in theimage object of the foreground which is captured when the foregroundobject is stationary, the pixel values of the consecutive pixelspositioned at the boundary of the image object corresponding to themoving direction of the foreground object, the number of pixels beingtwo times the amount of movement v, must be constant.

[0407] As discussed above, the mixture ratio α of the pixels belongingto the covered background area is calculated by equation (21), and themixture ratio α of the pixels belonging to the uncovered background areais calculated by equation (22).

α=(C−N)/(P−N)  (21)

α=(C−P)/(N−P)  (22)

[0408]FIG. 55 is a block diagram illustrating the configuration of theestimated-mixture-ratio processor 401. A frame memory 421 stores aninput image in the units of frames, and supplies the frame subsequent tothe frame which is input as the input image to a frame memory 422 and amixture-ratio calculator 423.

[0409] A frame memory 422 stores an input image in the units of frames,and supplies the frame subsequent to the frame supplied from the framememory 421 to the mixture-ratio calculator 423.

[0410] Accordingly, when frame #n+1 is input into the mixture-ratiocalculator 423 as the input image, the frame memory 421 supplies frame#n to the mixture-ratio calculator 423, and the frame memory 422supplies frame #n−1 to the mixture-ratio calculator 423.

[0411] The mixture-ratio calculator 423 calculates the estimated mixtureratio of interest by solving equation (21) based on the pixel value C ofthe pixel of interest in frame #n, the pixel value N of the pixel inframe #n+1 located at the position corresponding to the position of thepixel of interest, and the pixel value P of the pixel #n−1 located atthe position corresponding to the position of the pixel of interest, andoutputs the calculated estimated mixture ratio. For example, when thebackground is stationary, the mixture-ratio calculator 423 calculatesthe estimated mixture ratio of the pixel of interest based on the pixelvalue C of the pixel of interest in frame #n, the pixel value N of thepixel in frame #n+1 located at the same position as the pixel ofinterest, and the pixel value P of the pixel in frame #n−1 located atthe same position as the pixel of interest, and outputs the calculatedestimated mixture ratio.

[0412] In this manner, the estimated-mixture-ratio calculator 401calculates the estimated mixture ratio based on the input image, andsupplies it to the mixture-ratio determining portion 403.

[0413] The estimated-mixture-ratio calculator 401 calculates theestimated mixture ratio of the pixel of interest by solving equation(21). The operation of the estimated-mixture-ratio calculator 402 issimilar to that of the estimated-mixture-ratio calculator 401, exceptthat the estimated-mixture-ratio calculator 402 calculates a differentestimated mixture ratio of the pixel of interest by solving equation(22). Thus, an explanation of the estimated-mixture-ratio calculator 402is omitted.

[0414]FIG. 56 illustrates an example of the estimated mixture ratiocalculated by the estimated-mixture-ratio processor 401. The estimatedmixture ratio shown in FIG. 56 is the result represented by one linewhen the amount of movement v of the foreground object moving withconstant velocity is 11.

[0415] It is seen, as shown in FIG. 50, that the estimated mixture ratiochanges almost linearly in the mixed area.

[0416] Referring back to FIG. 49, the mixture-ratio determining portion403 sets the mixture ratio α based on the area information supplied fromthe area specifying unit 103 and indicating to which of the foregroundarea, the background area, the covered background area, or the uncoveredbackground area the pixel for which the mixture ratio α is to becalculated belongs. The mixture-ratio determining portion 403 sets themixture ratio α to 0 when the corresponding pixel belongs to theforeground area, and sets the mixture ratio α to 1 when thecorresponding pixel belongs to the background area. When thecorresponding pixel belongs to the covered background area, themixture-ratio determining portion 403 sets the mixture ratio α to theestimated mixture ratio supplied from the estimated-mixture-ratioprocessor 401. When the corresponding pixel belongs to the uncoveredbackground area, the mixture-ratio determining portion 403 sets themixture ratio α to the estimated mixture ratio supplied from theestimated-mixture-ratio processor 402. The mixture-ratio determiningportion 403 outputs the mixture ratio α which has been set based on thearea information.

[0417]FIG. 57 is a block diagram illustrating another configuration ofthe mixture-ratio calculator 104. A selector 441 supplies a pixelbelonging to the covered background area and the corresponding pixels inthe previous and subsequent frames to an estimated-mixture-ratioprocessor 442 based on the area information supplied from the areaspecifying unit 103. The selector 441 supplies a pixel belonging to theuncovered background area and the corresponding pixels in the previousand subsequent frames to an estimated-mixture-ratio processor 443 basedon the area information supplied from the area specifying unit 103.

[0418] The estimated-mixture-ratio processor 442 calculates theestimated mixture ratio of the pixel of interest belonging to thecovered background area by the calculation expressed in equation (21)based on the pixel values input from the selector 441, and supplies thecalculated estimated mixture ratio to a selector 444.

[0419] The estimated-mixture-ratio processor 443 calculates theestimated mixture ratio of the pixel of interest belonging to theuncovered background area by the calculation expressed in equation (22)based on the pixel values input from the selector 441, and supplies thecalculated estimated mixture ratio to the selector 444.

[0420] Based on the area information supplied from the area specifyingunit 103, the selector 444 selects the estimated mixture ratio 0 andsets it as the mixture ratio α when the pixel of interest belongs to theforeground area, and selects the estimated mixture ratio 1 and sets itas the mixture ratio α when the pixel of interest belongs to thebackground area. When the pixel of interest belongs to the coveredbackground area, the selector 444 selects the estimated mixture ratiosupplied from the estimated-mixture-ratio processor 442 and sets it asthe mixture ratio α. When the pixel of interest belongs to the uncoveredbackground area, the selector 444 selects the estimated mixture ratiosupplied from the estimated-mixture-ratio processor 443 and sets it asthe mixture ratio α. The selector 444 then outputs the mixture ratio αwhich has been selected and set based on the area information.

[0421] As discussed above, the mixture-ratio calculator 104 configuredas shown in FIG. 57 is able to calculate the mixture ratio α for eachpixel contained in the image, and outputs the calculated mixture ratioα.

[0422] The calculation processing for the mixture ratio a performed bythe mixture-ratio calculator 104 configured as shown in FIG. 49 isdiscussed below with reference to the flowchart of FIG. 58. In stepS401, the mixture-ratio calculator 104 obtains area information suppliedfrom the area specifying unit 103. In step S402, theestimated-mixture-ratio processor 401 executes the processing forestimating the mixture ratio by using a model corresponding to a coveredbackground area, and supplies the estimated mixture ratio to themixture-ratio determining portion 403. Details of the processing forestimating the mixture ratio are discussed below with reference to theflowchart of FIG. 59.

[0423] In step S403, the estimated-mixture-ratio processor 402 executesthe processing for estimating the mixture ratio by using a modelcorresponding to an uncovered background area, and supplies theestimated mixture ratio to the mixture-ratio determining portion 403.

[0424] In step S404, the mixture-ratio calculator 104 determines whetherthe mixture ratios have been estimated for the whole frame. If it isdetermined that the mixture ratios have not yet been estimated for thewhole frame, the process returns to step S402, and the processing forestimating the mixture ratio for the subsequent pixel is executed.

[0425] If it is determined in step S404 that the mixture ratios havebeen estimated for the whole frame, the process proceeds to step S405.In step S405, the mixture-ratio determining portion 403 sets the mixtureratio based on the area information supplied from the area specifyingunit 103 and indicating to which of the foreground area, the backgroundarea, the covered background area, or the uncovered background area thepixel for which the mixture ratio α is to be calculated belongs. Themixture-ratio determining portion 403 sets the mixture ratio α to 0 whenthe corresponding pixel belongs to the foreground area, and sets themixture ratio α to 1 when the corresponding pixel belongs to thebackground area. When the corresponding pixel belongs to the coveredbackground area, the mixture-ratio determining portion 403 sets theestimated mixture ratio supplied from the estimated-mixture-ratioprocessor 401 as the mixture ratio α. When the corresponding pixelbelongs to the uncovered background area, the mixture-ratio determiningportion 403 sets the estimated mixture ratio supplied from theestimated-mixture-ratio processor 402 as the mixture ratio α. Theprocessing is then completed.

[0426] As discussed above, the mixture-ratio calculator 104 is able tocalculate the mixture ratio α, which indicates a feature quantitycorresponding to each pixel, based on the area information supplied fromthe area specifying unit 103, and the input image.

[0427] The processing for calculating the mixture ratio a performed bythe mixture-ratio calculator 104 configured as shown in FIG. 57 issimilar to that discussed with reference to the flowchart of FIG. 58,and an explanation thereof is thus omitted.

[0428] A description is now given of, with reference to the flowchart ofFIG. 59, the mixture-ratio estimating processing by using a modelcorresponding to the covered background area in step S402 of FIG. 58.

[0429] In step S421, the mixture-ratio calculator 423 obtains the pixelvalue C of the pixel of interest in frame #n from the frame memory 421.

[0430] In step S422, the mixture-ratio calculator 423 obtains from theframe memory 422 the pixel value P of the pixel in frame #n−1corresponding to the pixel of interest contained in the input image.

[0431] In step S423, the mixture-ratio calculator 423 obtains the pixelvalue N of the pixel in frame #n+1 corresponding to the pixel ofinterest contained in the input image.

[0432] In step S424, the mixture-ratio calculator 423 calculates theestimated mixture ratio based on the pixel value C of the pixel ofinterest in frame #n, the pixel value P of the pixel in frame #n−1, andthe pixel value N of the pixel in frame #n+1.

[0433] In step S425, the mixture-ratio calculator 423 determines whetherthe processing for calculating the estimated mixture ratio is finishedfor the whole frame. If it is determined that the processing forcalculating the estimated mixture ratio is not finished for the wholeframe, the process returns to step S421, the processing for calculatingthe estimated mixture ratio for the subsequent pixel is repeated.

[0434] If it is determined in step S425 that the processing forcalculating the estimated mixture ratio for the whole frame is finished,the processing is completed.

[0435] As discussed above, the estimated-mixture-ratio processor 401 isable to calculate the estimated mixture ratio based on the input image.

[0436] The mixture-ratio estimating processing performed by using amodel corresponding to the uncovered background area in step S403 ofFIG. 58 is similar to the processing indicated by the flowchart of FIG.59 performed by using a model corresponding to the uncovered backgroundarea, and an explanation thereof is thus omitted.

[0437] The estimated-mixture-ratio processor 442 and theestimated-mixture-ratio processor 443 shown in FIG. 57 calculate theestimated mixture ratios by performing processing similar to that of theflowchart of FIG. 59, and an explanation thereof is thus omitted.

[0438] The embodiment has been described, assuming that the objectcorresponding to the background is stationary. However, theabove-described processing for determining the mixture ratio α can beapplied even if the image corresponding to the background area containsmotion. For example, if the image corresponding to the background areais uniformly moving, the estimated-mixture-ratio processor 401 shiftsthe overall image in accordance with the motion of the background, andperforms processing in a manner similar to the case in which the objectcorresponding to the background is stationary. If the imagecorresponding to the background area contains locally different motions,the estimated-mixture-ratio processor 401 selects the pixelscorresponding to the motions as the corresponding pixels belonging tothe mixed area, and executes the above-described processing.

[0439] The estimated-mixture ratio calculator 104 may execute only themixture-ratio estimating processing for all the pixels by using a modelcorresponding to the covered background area so as to output thecalculated estimated mixture ratio as the mixture ratio α. In this case,the mixture ratio α indicates the ratio of the background components forthe background components of the pixels belonging to the coveredbackground area, and indicates the ratio of the foreground componentsfor the pixels belonging to the uncovered background area. For thepixels belonging to the uncovered background area, the absolute value ofthe difference between the mixture ratio α and 1 is calculated, and thecalculated absolute value is set as the mixture ratio α. Then, thesignal processor 12 is able to determine the mixture ratio α indicatingthe ratio of the background components for the pixels belonging to theuncovered background area.

[0440] Similarly, the mixture-ratio calculator 104 may execute only themixture-ratio estimating processing for all the pixels by using a modelcorresponding to the uncovered background area so as to output thecalculated estimated mixture ratio as the mixture ratio α.

[0441] Another processing performed by the mixture-ratio calculator 104is discussed below.

[0442] The mixture ratio α linearly changes in accordance with a changein the position of the pixels because the object corresponding to theforeground is moving with constant velocity. By utilizing thischaracteristic, an equation in which the mixture ratio α and the sum fof the foreground components are approximated in the spatial directioncan hold true. By utilizing a plurality of sets of the pixel values ofthe pixels belonging to the mixed area and the pixel values of thepixels belonging to the background area, the mixture ratio α can becalculated by solving the equations in which the mixture ratio α and thesum f of the foreground components are approximated.

[0443] When a change in the mixture ratio α is approximated as astraight line, the mixture ratio α can be expressed by equation (23).

α=il+p  (23)

[0444] In equation (23), i indicates the spatial index when the positionof the pixel of interest is set to 0, l designates the gradient of thestraight line of the mixture ratio α, and p designates the intercept ofthe straight line of the mixture ratio α and also indicates the mixtureratio α of the pixel of interest. In equation (23), the index i isknown, and the gradient l and the intercept p are unknown.

[0445] The relationship among the index i, the gradient l, and theintercept p is shown in FIG. 60.

[0446] By approximating the mixture ratio α as equation (23), aplurality of different mixture ratios a for a plurality of pixels can beexpressed by two variables. In the example shown in FIG. 60, the fivemixture ratios for five pixels are expressed by the two variables, i.e.,the gradient l and the intercept p.

[0447] When the mixture ratio α is approximated in the plane shown inFIG. 61, equation (23) is expanded into the plane by considering themovement v corresponding to the two directions, i.e., the horizontaldirection and the vertical direction of the image, and the mixture ratioα can be expressed by equation (24).

α=jm+kq+p  (24)

[0448] In equation (24), j is the index in the horizontal direction andk is the index in the vertical direction when the position of the pixelof interest is 0. In equation (24), m designates the horizontal gradientof the mixture ratio α in the plane, and q indicates the verticalgradient of the mixture ratio α in the plane. In equation (24), pindicates the intercept of the mixture ratio α in the plane.

[0449] For example, in frame #n shown in FIG. 51, equations (25) through(27) can hold true for C05 through C07, respectively.

C05=α05·B05/v+f05  (25)

C06=α06·B06/v+f06  (26)

C07=α07·B07/v+f07  (27)

[0450] Assuming that the foreground components positioned in closeproximity with each other are equal to each other, i.e., that F01through F03 are equal, equation (28) holds true by replacing F01 throughF03 by fc.

f(x)=(1−α(x))·Fc  (28)

[0451] In equation (28), x indicates the position in the spatialdirection.

[0452] When α(x) is replaced by equation (24), equation (28) can beexpressed by equation (29). $\begin{matrix}\begin{matrix}{{f(x)} = {( {1 - ( {{jm} + {kq} + p} )} ) \cdot {Fc}}} \\{= {{j \cdot ( {{- m} \cdot {Fc}} )} + {k \cdot ( {{- q} \cdot {Fc}} )} + ( {( {1 - p} ) \cdot {Fc}} )}} \\{= {{js} + {kt} + u}}\end{matrix} & (29)\end{matrix}$

[0453] In equation (29), (−m·Fc), (−q·Fc), and (1−p)·Fc are replaced, asexpressed by equations (30) through (32), respectively.

s=−m·Fc  (30)

t=−q·Fc  (31)

u=(1−p)·Fc  (32)

[0454] In equation (29), j is the index in the horizontal direction andk is the index in the vertical direction when the position of the pixelof interest is 0.

[0455] As discussed above, since it can be assumed that the objectcorresponding to the foreground is moving with constant velocity withinthe shutter period, and that the foreground components positioned inclose proximity with each other are uniform, the sum of the foregroundcomponents can be approximated by equation (29).

[0456] When the mixture ratio α is approximated by a straight line, thesum of the foreground components can be expressed by equation (33).

f(x)=is+u  (33)

[0457] By replacing the mixture ratio α and the sum of the foregroundcomponents in equation (13) by using equations (24) and (29), the pixelvalue M can be expressed by equation (34). $\begin{matrix}\begin{matrix}{M = {{( {{jm} + {kq} + p} ) \cdot B} + {js} + {kt} + u}} \\{= {{{jB} \cdot m} + {{kB} \cdot q} + {B \cdot p} + {j \cdot s} + {k \cdot t} + u}}\end{matrix} & (34)\end{matrix}$

[0458] In equation (34), unknown variables are six factors, such as thehorizontal gradient m of the mixture ratio α in the plane, the verticalgradient q of the mixture ratio α in the plane, and the intercepts ofthe mixture ratio α in the plane, p, s, t, and u.

[0459] The pixel value M and the pixel value B are set in equation (34)in accordance with the pixels close to the pixel of interest, and then,a plurality of equations in which the pixel value M and the pixel valueB are set are solved by the method of least squares, thereby calculatingthe mixture ratio α.

[0460] It is now assumed, for example, that the horizontal index j ofthe pixel of interest is set to 0, and the vertical index k of the pixelof interest is set to 0. In this case, when the pixel value M or thepixel value B is set in the normal equation expressed by equation (34)for 3×3 pixels located in the proximity with the pixel of interest,equations (35) through (43) are obtained.

M _(−1,−1)=(−1)·B _(−1,−1) ·m+(−1)·B _(−1,−1) ·q+B _(−1,−1)·p+(−1)·s+(−1)·t+u  (35)

M _(0,−1)=(0)·B _(0,−1) ·m+(−1)·B _(0,−1) q+B _(0,−1)·p+(0)·s+(−1)·t+u  (36)

M _(+1,−1)=(+1)·B _(+1,−1) ·m+(−1)·B _(+1,−1) ·q+B _(+1,−1)·p+(+1)·s+(−1)·t+u  (37)

M _(−1,0)=(−1)·B _(−1,0) ·m+(0)·B _(−1,0) ·q+B _(−1,0)·p+(−1)·s+(0)·t+u  (38)

M _(0,0)=(0)·B _(0,0) ·m+(0)·B _(0,0) ·q+B _(0,0) ·p+(0)·s+(0)·t+u  (39)

M _(+1,0)=(+1)·B _(+1,0) ·m+(0)·B_(+1,0) ·q+B _(+1,0)·p+(+1)·s+(0)·t+u  (40)

M _(−1,+1)=(−1)·B _(−1,+1) ·m+(+1)·B _(−1,+1) ·q+B _(−1,+1)·p+(−1)·s+(+1)·t+u  (41)

M _(0,+1)=(0)·B _(0,+1) ·m+(+1)·B _(0,+1) ·q+B _(0,+1)·p+(0)·s+(+1)·t+u  (42)

M _(+1,+1)=(+1)·B _(+1,+1) ·m+(+1)·B _(+1,+1) ·q+B _(+1,+1)·p+(+1)·s+(+1)·t+u  (43)

[0461] Since the horizontal index j of the pixel of interest is 0, andthe vertical index k of the pixel of interest is 0, the mixture ratio αof the pixel of interest is equal to the value when j is 0 and k is 0 inequation (24), i.e., the mixture ratio α is equal to the intercept p inequation (24).

[0462] Accordingly, based on the nine equations, i.e., equations (35)through (43), the horizontal gradient m, the vertical gradient q, andthe intercepts p, s, t, and u are calculated by the method of leastsquares, and the intercept p is output as the mixture ratio α.

[0463] A specific procedure for calculating the mixture ratio α byapplying the method of least squares is as follows.

[0464] When the index i and the index k are expressed by a single indexx, the relationship among the index i, the index k, and the index x canbe expressed by equation (44).

x=(j+1)·3+(k+1)  (44)

[0465] It is now assumed that the horizontal gradient m, the verticalgradient q, and the intercepts p, s, t, and u are expressed by variablesw0, w1, w2, w3, w4, and w5, respectively, and jB, kB, B, j, k and l areexpressed by a0, a1, a2, a3, a4, and a5, respectively. In considerationof the error ex, equations (35) through (43) can be modified$\begin{matrix}{{Mx} = {{\sum\limits_{y = 0}^{5}{{ay} \cdot {wy}}} + {ex}}} & (45)\end{matrix}$

[0466] In equation (45), x is any one of the integers from 0 to 8.

[0467] Equation (46) can be found from equation (45). $\begin{matrix}{{ex} = {{Mx} - {\sum\limits_{y = 0}^{5}{{ay} \cdot {wy}}}}} & (46)\end{matrix}$

[0468] Since the method of least squares is applied, the square sum E ofthe error is defined as follows, as expressed by equation (47).$\begin{matrix}{E = {\sum\limits_{x = 0}^{8}{ex}^{2}}} & (47)\end{matrix}$

[0469] In order to minimize the error, the partial differential value ofthe variable Wv with respect to the square sum E of the error should be0. v is any one of the integers from 0 to 5. Thus, wy is determined sothat equation (48) is satisfied. $\begin{matrix}\begin{matrix}{\frac{\partial E}{\partial{Wv}} = {2 \cdot {\sum\limits_{x = 0}^{8}{{ex} \cdot \frac{\partial{ex}}{\partial{Wv}}}}}} \\{= {{2 \cdot {\sum\limits_{x = 0}^{8}{{ex} \cdot {av}}}} = 0}}\end{matrix} & (48)\end{matrix}$

[0470] By substituting equation (46) into equation (48), equation (49)is obtained. $\begin{matrix}{{\sum\limits_{x = 0}^{8}( {{av} \cdot {\sum\limits_{y = 0}^{5}{{ay} \cdot {Wy}}}} )} = {\sum\limits_{x = 0}^{8}{{av} \cdot {Mx}}}} & (49)\end{matrix}$

[0471] For example, the sweep-out method (Gauss-Jordan elimination) isapplied to the normal equations consisting of six equations obtained bysubstituting one of the integers from 0 to 5 into v in equation (49),thereby obtaining wy. As stated above, w0 is the horizontal gradient m,w1 is the vertical gradient q, w2 is the intercept p, w3 is s, w4 is t,and w5 is u.

[0472] As discussed above, by applying the method of least squares tothe equations in which the pixel value M and the pixel value B are set,the horizontal gradient m, the vertical gradient q, and the interceptsp, s, t, and u can be determined.

[0473] The intercept p is the mixture ratio α when indexes i and k are0, i.e., the intercept p is located at the center position. Thus, theintercept P is output.

[0474] A description has been given with reference to equations (35)through (43), by assuming that the pixel value of the pixel contained inthe mixed area is M, and the pixel value of the pixel contained in thebackground area is B. In this case, it is necessary to set normalequations for each of the cases where the pixel of interest is containedin the covered background area, or the pixel of interest is contained inthe uncovered background area.

[0475] For example, if the mixture ratio α of the pixel contained in thecovered background area in frame #n shown in FIG. 51 is determined, C04through C08 of the pixels in frame #n and the pixel values P04 throughP08 of the pixels in frame #n−1 are set in the normal equations.

[0476] If the mixture ratio α of the pixels contained in the uncoveredbackground area in frame #n shown in FIG. 52 is determined, C28 throughC32 of the pixels in frame #n and the pixel values N28 through N32 ofthe pixels in frame #n+1 are set in the normal equations.

[0477] Moreover, if, for example, the mixture ratio α of the pixelcontained in the covered background area shown in FIG. 62 is calculated,the following equations (50) through (58) are set. The pixel value ofthe pixel for which the mixture ratio α is calculated is Mc5.

Mc1=(−1)·Bc1·m+(−1)·Bc1·q+Bc1·p+(−1)·s+(−1)·t+u  (50)

Mc2=(0)·Bc2·m+(−1)·Bc2·q+Bc2·p+(0)·s+(−1)·t+u  (51)

Mc3=(+1)·Bc3·m+(−1)·Bc3·q+Bc3·p+(+1)·s+(−1)·t+u  (53)

Mc4=(−1)·Bc4·m+(0)·Bc4·q+Bc4·p+(−1)·s+(0)·t+u  (53)

Mc5=(0)·Bc5·m+(0)·Bc5·q+Bc5·p+(0)·s+(0)·t+u  (54)

Mc6=(+1)·Bc6·m+(0)·Bc6·q+Bc6·p+(+1)·s+(0)·t+u  (55)

Mc7=(−1)·Bc7·m+(+1)·Bc7·q+Bc7·p+(−1)·s+(+1)·t+u  (56)

Mc8=(0)·Bc8·m+(+1)·Bc8·q+Bc8·p+(0)·s+(+1)·t+u  (57)

Mc9=(+1)·Bc9·m+(+1)·Bc9·q+Bc9·p+(+1)·s+(+1)·t+u  (58)

[0478] When the mixture ratio α of the pixel contained in the coveredbackground area in frame #n is calculated, the pixel values Bc1 throughBc9 of the pixels of the background area in frame #n−1 in equations (50)through (58), respectively, corresponding to the pixels in frame #n areused.

[0479] If, for example, the mixture ratio α of the pixel contained inthe uncovered background area shown in FIG. 62 is calculated, thefollowing equations (59) through (67) are set. The pixel value of thepixel for which the mixture ratio α is calculated is Mu5.

Mu1=(−1)·Bu1·m+(−1)·Bu1·q+Bu1·p+(−1)·s+(−1)·t+u  (59)

Mu2=(0)·Bu2·m+(−1)·Bu2·q+Bu2·p+(0)·s+(−1)·t+u  (60)

Mu3=(+1)·Bu3·m+(−1)·Bu3·q+Bu3·p+(+1)·s+(−1)·t+u  (61)

Mu4=(−1)·Bu4·m+(0)·Bu4·q+Bu4·p+(−1)·s+(0)·t+u  (62)

Mu5=(0)·Bu5·m+(0)·Bu5·q+Bu5·p+(0)·s+(0)·t+u  (63)

Mu6=(+1)·Bu6·m+(0)·Bu6·q+Bu6·p+(+1)·s+(0)·t+u  (64)

Mu7=(−1)·Bu7·m+(+1)·Bu7·q+Bu7·p+(−1)·s+(+1)·t+u  (65)

Mu8=(0)·Bu8·m+(+1)·Bu8·q+Bu8·p+(0)·s+(+1)·t+u  (66)

Mu9=(+1)·Bu9·m+(+1)·Bu9·q+Bu9·p+(+1)·s+(+1)·t+u  (67)

[0480] When the mixture ratio α of the pixel contained in the uncoveredbackground area in frame #n is calculated, the pixel values Bu1 throughBu9 of the pixels of the background area in frame #n+1 in equations (59)through (67), respectively, corresponding to the pixels in frame #n areused.

[0481]FIG. 63 is a block diagram illustrating the configuration of theestimated-mixture-ratio processor 401. An image input into theestimated-mixture-ratio processor 401 is supplied to a delay portion 501and an adder 502.

[0482] A delay circuit 221 delays the input image for one frame, andsupplies the image to the adder 502. When frame #n is supplied as theinput image to the adder 502, the delay circuit 221 supplies frame #n−1to the adder 502.

[0483] The adder 502 sets the pixel value of the pixel adjacent to thepixel for which the mixture ratio α is calculated, and the pixel valueof frame #n−1 in the normal equation. For example, the adder 502 setsthe pixel values Mc1 through Mc9 and the pixel values Bc1 through Bc9 inthe normal equations based on equations (50) through (58), respectively.The adder 502 supplies the normal equations in which the pixel valuesare set to a calculator 503.

[0484] The calculator 503 determines the estimated mixture ratio bysolving the normal equations supplied from the adder 502 by, forexample, a sweep-out method, and outputs the determined estimatedmixture ratio.

[0485] In this manner, the estimated-mixture-ratio processor 401 is ableto calculate the estimated mixture ratio based on the input image, andsupplies it to the mixture-ratio determining portion 403.

[0486] The estimated-mixture-ratio processor 402 is configured similarto the estimated-mixture-ratio processor 401, and an explanation thereofis thus omitted.

[0487]FIG. 64 illustrates an example of the estimated mixture ratiocalculated by the estimated-mixture-ratio processor 401. The estimatedmixture ratio shown in FIG. 64 is the result represented by one line andobtained by performing the calculation by generating equations in unitsof 7×7-pixel blocks when the movement v of the foreground correspondingto the object moving with constant velocity is 11.

[0488] The estimated mixture ratio changes almost linearly in the mixedarea, as shown in FIG. 50.

[0489] The mixture-ratio determining portion 403 sets the mixture ratiobased on the area information supplied from the area specifying unit 101and indicating to which of-the foreground area, the background area, thecovered background area, or the uncovered background area the pixel forwhich the mixture ratio is to be calculated belongs. The mixture-ratiodetermining portion 403 sets the mixture ratio to 0 when thecorresponding pixel belongs to the foreground area, and sets the mixtureratio to 1 when the corresponding pixel belongs to the background area.When the corresponding pixel belongs to the covered background area, themixture-ratio determining portion 403 sets the mixture ratio to theestimated mixture ratio supplied from the estimated-mixture-ratioprocessor 401. When the corresponding pixel belongs to the uncoveredbackground area, the mixture-ratio determining portion 403 sets themixture ratio to the estimated mixture ratio supplied from theestimated-mixture-ratio processor 402. The mixture-ratio determiningportion 403 outputs the mixture ratio which has been set based on thearea information.

[0490] The mixture-ratio calculation processing performed by themixture-ratio calculator 102 when the estimated-mixture-ratio processor401 is configured as shown in FIG. 63 is discussed below with referenceto the flowchart of FIG. 65. In step S501, the mixture-ratio calculator102 obtains area information supplied from the area specifying unit 101.In step S502, the estimated-mixture-ratio processor 401 executes theprocessing for estimating the mixture ratio by using a modelcorresponding to a covered background area, and supplies the estimatedmixture ratio to the mixture-ratio determining portion 403. Details ofthe processing for estimating the mixture ratio are discussed below withreference to the flowchart of FIG. 66.

[0491] In step S503, the estimated-mixture-ratio processor 402 executesthe processing for estimating the mixture ratio by using a modelcorresponding to an uncovered background area, and supplies theestimated mixture ratio to the mixture-ratio determining portion 403.

[0492] In step S504, the mixture-ratio calculator 102 determines whetherthe mixture ratios have been estimated for the whole frame. If it isdetermined that the mixture ratios have not yet been estimated for thewhole frame, the process returns to step S502, and the processing forestimating the mixture ratio for the subsequent pixel is executed.

[0493] If it is determined in step S504 that the mixture ratios havebeen estimated for the whole frame, the process proceeds to step S505.In step S505, the mixture-ratio determining portion 403 sets the mixtureratio based on the area information supplied from the area specifyingunit 101 and indicating to which of the foreground area, the backgroundarea, the covered background area, or the uncovered background area thepixel for which the mixture ratio is to be calculated belongs. Themixture-ratio determining portion 403 sets the mixture ratio to 0 whenthe corresponding pixel belongs to the foreground area, and sets themixture ratio to 1 when the corresponding pixel belongs to thebackground area. When the corresponding pixel belongs to the coveredbackground area, the mixture-ratio determining portion 403 sets theestimated mixture ratio supplied from the estimated-mixture-ratioprocessor 401 as the mixture ratio. When the corresponding pixel belongsto the uncovered background area, the mixture-ratio determining portion403 sets the estimated mixture ratio supplied from theestimated-mixture-ratio processor 402 as the mixture ratio. Theprocessing is then completed.

[0494] As discussed above, the mixture-ratio calculator 102 is able tocalculate the mixture ratio α, which indicates a feature quantitycorresponding to each pixel, based on the area information supplied fromthe area specifying unit 101, and the input image.

[0495] By utilizing the mixture ratio α, it is possible to separate theforeground components and the background components contained in thepixel values while maintaining the information of motion blur containedin the image corresponding to the moving object.

[0496] By combining the images based on the mixture ratio α, it is alsopossible to create an image which contains correct motion blur thatcoincides with the speed of a moving object and which faithfullyreflects the real world.

[0497] A description is now given, with reference to the flowchart ofFIG. 66, of the mixture-ratio estimating processing by using a model ofthe covered background area in step S502 of FIG. 65.

[0498] In step S521, the adder 502 sets the pixel value contained in theinput image and the pixel value contained in the image supplied from thedelay circuit 221 in a normal equation corresponding to a model of thecovered background area.

[0499] In step S522, the estimated-mixture-ratio processor 401determines whether the setting of the target pixels is finished. If itis determined that the setting of the target pixels is not finished, theprocess returns to step S521, and the processing for setting the pixelvalues in the normal equation is repeated.

[0500] If it is determined in step S522 that the setting for the targetpixels is finished, the process proceeds to step S523. In step S523, acalculator 173 calculates the estimated mixture ratio based on thenormal equations in which the pixels values are set, and outputs thecalculated mixture ratio.

[0501] As discussed above, the estimated-mixture-ratio processor 401 isable to calculate the estimated mixture ratio based on the input image.

[0502] The mixture-ratio estimating processing by using a modelcorresponding to the uncovered background area in step S153 of FIG. 65is similar to the processing indicated by the flowchart of FIG. 66 byusing the normal equations corresponding to a model of the uncoveredbackground area, and an explanation thereof is thus omitted.

[0503] The embodiment has been described, assuming that the objectcorresponding to the background is stationary. However, theabove-described mixture-ratio calculation processing can be applied evenif the image corresponding to the background area contains motion. Forexample, if the image corresponding to the background area is uniformlymoving, the estimated-mixture-ratio processor 401 shifts the overallimage in accordance with this motion, and performs processing in amanner similar to the case in which the object corresponding to thebackground is stationary. If the image corresponding to the backgroundarea contains locally different motions, the estimated-mixture-ratioprocessor 401 selects the pixels corresponding to the motions as thepixels belonging to the mixed area, and executes the above-describedprocessing.

[0504] The foreground/background separator 105 is discussed below. FIG.67 is a block diagram illustrating an example of the configuration ofthe foreground/background separator 105. The input image supplied to theforeground/background separator 105 is supplied to a separating portion601, a switch 602, and a switch 604. The area information supplied fromthe area specifying unit 103 and indicating the information of thecovered background area and the uncovered background area is supplied tothe separating portion 601. The area information indicating theforeground area is supplied to the switch 602. The area informationindicating the background area supplied to the switch 604.

[0505] The mixture ratio α supplied from the mixture-ratio calculator104 is supplied to the separating portion 601.

[0506] The separating portion 601 separates the foreground componentsfrom the input image based on the area information indicating thecovered background area, the area information indicating the uncoveredbackground area, and the mixture ratio α, and supplies the separatedforeground components to a synthesizer 603. The separating portion 601also separates the background components from the input image, andsupplies the separated background components to a synthesizer 605.

[0507] The switch 602 is closed when a pixel corresponding to theforeground is input based on the area information indicating theforeground area, and supplies only the pixels corresponding to theforeground contained in the input image to the synthesizer 603.

[0508] The switch 604 is closed when a pixel corresponding to thebackground is input based on the area information indicating thebackground area, and supplies only the pixels corresponding to thebackground contained in the input image to the synthesizer 605.

[0509] The synthesizer 603 synthesizes a foreground component imagebased on the foreground components supplied from the separating portion601 and the pixels corresponding to the foreground supplied from theswitch 602, and outputs the synthesized foreground component image.Since the foreground area and the mixed area do not overlap, thesynthesizer 603 applies, for example, logical OR to the foregroundcomponents and the foreground pixels, thereby synthesizing theforeground component image.

[0510] In the initializing processing executed at the start of thesynthesizing processing for the foreground component image, thesynthesizer 603 stores an image whose pixel values are all 0 in abuilt-in frame memory. Then, in the synthesizing processing for theforeground component image, the synthesizer 603 stores the foregroundcomponent image (overwrites the previous image by the foregroundcomponent image). Accordingly, 0 is stored in the pixels correspondingto the background area in the foreground component image output from thesynthesizer 603.

[0511] The synthesizer 605 synthesizes a background component imagebased on the background components supplied from the separating portion601 and the pixels corresponding to the background supplied from theswitch 604, and outputs the synthesized background component image.Since the background area and the mixed area do not overlap, thesynthesizer 605 applies, for example, logical OR to the backgroundcomponents and the background pixels, thereby synthesizing thebackground component image.

[0512] In the initializing processing executed at the start of thesynthesizing processing for the background component image, thesynthesizer 605 stores an image whose pixel values are all 0 in abuilt-in frame memory. Then, in the synthesizing processing for thebackground component image, the synthesizer 605 stores the backgroundcomponent image (overwrites the previous image by the backgroundcomponent image). Accordingly, 0 is stored in the pixels correspondingto the foreground area in the background component image output from thesynthesizer 605.

[0513]FIG. 68A illustrates the input image input into theforeground/background separator 105 and the foreground component imageand the background component image output from the foreground/backgroundseparator 105. FIG. 68B illustrates a model obtained by expanding in thetime direction the pixels disposed in one line including the pixelsbelonging to the foreground area, the pixels belonging to the backgroundarea, and the pixels belonging to the mixed area corresponding to FIG.68A.

[0514] As shown in FIGS. 68A and 68B, the background component imageoutput from the foreground/background separator 105 consists of thepixels belonging to the background area and the background componentscontained in the pixels of the mixed area.

[0515] As shown in FIGS. 68A and 68B, the foreground component imageoutput from the foreground/background separator 105 consists of thepixel belonging to the foreground area and the foreground componentscontained in the pixels of the mixed area.

[0516] The pixel values of the pixels in the mixed area are separatedinto the background components and the foreground components by theforeground/background separator 105. The separated background componentsform the background component image together with the pixels belongingto the background area. The separated foreground components form theforeground component image together with the pixels belonging to theforeground area.

[0517] As discussed above, in the foreground component image, the pixelvalues of the pixels corresponding to the background area are set to 0,and significant pixel values are set in the pixels corresponding to theforeground area and the pixels corresponding to the mixed area.Similarly, in the background component image, the pixel values of thepixels corresponding to the foreground area are set to 0, andsignificant pixel values are set in the pixels corresponding to thebackground area and the pixels corresponding to the mixed area.

[0518] A description is given below of the processing executed by theseparating portion 601 for separating the foreground components and thebackground components from the pixels belonging to the mixed area.

[0519]FIG. 69 illustrates a model of an image indicating foregroundcomponents and background components in two frames including aforeground object moving from the left to the right in FIG. 69. In themodel of the image shown in FIG. 69, the amount of movement v is 4, andthe number of virtual divided portions is 4.

[0520] In frame #n, the leftmost pixel and the fourteenth througheighteenth pixels from the left consist of only the backgroundcomponents and belong to the background area. In frame #n, the secondthrough fourth pixels from the left contain the background componentsand the foreground components, and belong to the uncovered backgroundarea. In frame #n, the eleventh through thirteenth pixels from the leftcontain background components and foreground components, and belong tothe covered background area. In frame #n, the fifth through tenth pixelsfrom the left consist of only the foreground components, and belong tothe foreground area.

[0521] In frame #n+1, the first through fifth pixels from the left andthe eighteenth pixel from the left consist of only the backgroundcomponents, and belong to the background area. In frame #n+1, the sixththrough eighth pixels from the left contain background components andforeground components, and belong to the uncovered background area. Inframe #n+1, the fifteenth through seventeenth pixels from the leftcontain background components and foreground components, and belong tothe covered background area. In frame #n+1, the ninth through fourteenthpixels from the left consist of only the foreground components, andbelong to the foreground area.

[0522]FIG. 70 illustrates the processing for separating the foregroundcomponents from the pixels belonging to the covered background area. InFIG. 70, α1 through α18 indicate mixture ratios of the individual pixelsof frame #n. In FIG. 70, the fifteenth through seventeenth pixels fromthe left belong to the covered background area.

[0523] The pixel value C15 of the fifteenth pixel from the left in frame#n can be expressed by equation (68): $\begin{matrix}\begin{matrix}{{C15} = {{{B15}/v} + {{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{{\alpha 15} \cdot {B15}} + {{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{{\alpha 15} \cdot {P15}} + {{F09}/v} + {{F08}/v} + {{F07}/v}}}\end{matrix} & (68)\end{matrix}$

[0524] where α15 indicates the mixture ratio of the fifteenth pixel fromthe left in frame #n, and P15 designates the pixel value of thefifteenth pixel from the left in frame #n−1.

[0525] The sum f15 of the foreground components of the fifteenth pixelfrom the left in frame #n can be expressed by equation (69) based onequation (68). $\begin{matrix}\begin{matrix}{{f15} = {{{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{C15} - {{\alpha 15} \cdot {P15}}}}\end{matrix} & (69)\end{matrix}$

[0526] Similarly, the sum f16 of the foreground components of thesixteenth pixel from the left in frame #n can be expressed by equation(70), and the sum f17 of the foreground components of the seventeenthpixel from the left in frame #n can be expressed by equation (71).

f16=C16−α16·P16  (70)

f17=C17−α17·P17  (71)

[0527] In this manner, the foreground components fc contained in thepixel value C of the pixel belonging to the covered background area canbe expressed by equation (72):

fc=C−α·P  (72)

[0528] where P designates the pixel value of the corresponding pixel inthe previous frame.

[0529]FIG. 71 illustrates the processing for separating the foregroundcomponents from the pixels belonging to the uncovered background area.In FIG. 71, α1 through α18 indicate mixture ratios of the individualpixels of frame #n. In FIG. 71, the second through fourth pixels fromthe left belong to the uncovered background area.

[0530] The pixel value C02 of the second pixel from the left in frame #ncan be expressed by equation (73): $\begin{matrix}\begin{matrix}{{C02} = {{{B02}/v} + {{B02}/v} + {{B02}/v} + {{F01}/v}}} \\{= {{{\alpha 2} \cdot {B02}} + {{F01}/v}}} \\{= {{{\alpha 2} \cdot {N02}} + {{F01}/v}}}\end{matrix} & (73)\end{matrix}$

[0531] where α2 indicates the mixture ratio of the second pixel from theleft in frame #n, and N02 designates the pixel value of the second pixelfrom the left in frame #n+1.

[0532] The sum f02 of the foreground components of the second pixel fromthe left in frame #n can be expressed by equation (74) based on equation(73). $\begin{matrix}\begin{matrix}{{f02} = {{F01}/v}} \\{= {{C02} - {{\alpha 2} \cdot {N02}}}}\end{matrix} & (74)\end{matrix}$

[0533] Similarly, the sum f03 of the foreground components of the thirdpixel from the left in frame #n can be expressed by equation (75), andthe sum f04 of the foreground components of the fourth pixel from theleft in frame #n can be expressed by equation (76).

f03=C03−α3·N03  (75)

f04=C04−α4·N04  (76)

[0534] In this manner, the foreground components fu contained in thepixel value C of the pixel belonging to the uncovered background areacan be expressed by equation (77):

fu=C−α·N  (77)

[0535] where N designates the pixel value of the corresponding pixel inthe subsequent frame.

[0536] As discussed above, the separating portion 601 is able toseparate the foreground components from the pixels belonging to themixed area and the background components from the pixels belonging tothe mixed area based on the information indicating the coveredbackground area and the information indicating the uncovered backgroundarea contained in the area information, and the mixture ratio a for eachpixel.

[0537]FIG. 72 is a block diagram illustrating an example of theconfiguration of the separating portion 601 for executing theabove-described processing. An image input into the separating portion601 is supplied to a frame memory 621, and the area informationindicating the covered background area and the uncovered background areasupplied from the mixture-ratio calculator 104 and the mixture ratio aare supplied to a separation processing block 622.

[0538] The frame memory 621 stores the input images in units of frames.When a frame to be processed is frame #n, the frame memory 621 storesframe #n−1, which is the frame one frame before frame #n, frame #n, andframe #n+1, which is the frame one frame after frame #n.

[0539] The frame memory 621 supplies the corresponding pixels in frame#n−1, frame #n, and frame #n+1 to the separation processing block 622.

[0540] The separation processing block 622 applies the calculationsdiscussed with reference to FIGS. 70 and 71 to the pixel values of thecorresponding pixels in frame #n−1, frame #n, and frame #n+1 suppliedfrom the frame memory 621 based on the area information indicating thecovered background area and the uncovered background area and themixture ratio α so as to separate the foreground components and thebackground components from the pixels belonging to the mixed area inframe #n, and supplies them to a frame memory 623.

[0541] The separation processing block 622 is formed of an uncoveredarea processor 631, a covered area processor 632, a synthesizer 633, anda synthesizer 634.

[0542] A multiplier 641 of the uncovered area processor 631 multipliesthe pixel value of the pixel in frame #n+1 supplied from the framememory 621 by the mixture ratio α, and outputs the resulting pixel valueto a switch 642. The switch 642 is closed when the pixel of frame #n(corresponding to the pixel in frame #n+1) supplied from the framememory 621 belongs to the uncovered background area, and supplies thepixel value multiplied by the mixture ratio a supplied from themultiplier 641 to a calculator 643 and the synthesizer 634. The valueobtained by multiplying the pixel value of the pixel in frame #n+1 bythe mixture ratio a output from the switch 642 is equivalent to thebackground components of the pixel value of the corresponding pixel inframe #n.

[0543] The calculator 643 subtracts the background components suppliedfrom the switch 642 from the pixel value of the pixel in frame #nsupplied from the frame memory 621 so as to obtain the foregroundcomponents. The calculator 643 supplies the foreground components of thepixel in frame #n belonging to the uncovered background area to thesynthesizer 633.

[0544] A multiplier 651 of the covered area processor 632 multiplies thepixel value of the pixel in frame #n−1 supplied from the frame memory621 by the mixture ratio α, and outputs the resulting pixel value to aswitch 652. The switch 652 is closed when the pixel of frame #n(corresponding to the pixel in frame #n−1) supplied from the framememory 621 belongs to the covered background area, and supplies thepixel value multiplied by the mixture ratio α supplied from themultiplier 651 to a calculator 653 and the synthesizer 634. The valueobtained by multiplying the pixel value of the pixel in frame #n−1 bythe mixture ratio α output from the switch 652 is equivalent to thebackground components of the pixel value of the corresponding pixel inframe #n.

[0545] The calculator 653 subtracts the background components suppliedfrom the switch 652 from the pixel value of the pixel in frame #nsupplied from the frame memory 621 so as to obtain the foregroundcomponents. The calculator 653 supplies the foreground components of thepixel in frame #n belonging to the covered background area to thesynthesizer 633.

[0546] The synthesizer 633 combines the foreground components of thepixels belonging to the uncovered background area and supplied from thecalculator 643 with the foreground components of the pixels belonging tothe covered background area and supplied from the calculator 653, andsupplies the synthesized foreground components to the frame memory 623.

[0547] The synthesizer 634 combines the background components of thepixels belonging to the uncovered background area and supplied from theswitch 642 with the background components of the pixels belonging to thecovered background area and supplied from the switch 652, and suppliesthe synthesized background components to the frame memory 623.

[0548] The frame memory 623 stores the foreground components and thebackground components of the pixels in the mixed area of frame #nsupplied from the separation processing block 622.

[0549] The frame memory 623 outputs the stored foreground components ofthe pixels in the mixed area in frame #n and the stored backgroundcomponents of the pixels in the mixed area in frame #n.

[0550] By utilizing the mixture ratio α, which indicates the featurequantity, the foreground components and the background componentscontained in the pixel values can be completely separated.

[0551] The synthesizer 603 combines the foreground components of thepixels in the mixed area in frame #n output from the separating portion601 with the pixels belonging to the foreground area so as to generate aforeground component image. The synthesizer 605 combines the backgroundcomponents of the pixels in the mixed area in frame #n output from theseparating portion 601 with the pixels belonging to the background areaso as to generate a background component image.

[0552]FIG. 73A illustrates an example of the foreground component imagecorresponding to frame #n in FIG. 69. The leftmost pixel and thefourteenth pixel from the left consist of only the background componentsbefore the foreground and the background are separated, and thus, thepixel values are set to 0.

[0553] The second and fourth pixels from the left belong to theuncovered background area before the foreground and the background areseparated. Accordingly, the background components are set to 0, and theforeground components are maintained. The eleventh through thirteenthpixels from the left belong to the covered background area before theforeground and the background are separated. Accordingly, the backgroundcomponents are set to 0, and the foreground components are maintained.The fifth through tenth pixels from the left consist of only theforeground components, which are thus maintained.

[0554]FIG. 73B illustrates an example of the background component imagecorresponding to frame #n in FIG. 69. The leftmost pixel and thefourteenth pixel from the left consist of only the background componentsbefore the foreground and the background are separated, and thus, thebackground components are maintained.

[0555] The second through fourth pixels from the left belong to theuncovered background area before the foreground and the background areseparated. Accordingly, the foreground components are set to 0, and thebackground components are maintained. The eleventh through thirteenthpixels from the left belong to the covered background area before theforeground and the background are separated. Accordingly, the foregroundcomponents are set to 0, and the background components are maintained.The fifth through tenth pixels from the left consist of only theforeground components, and thus, the pixel values are set to 0.

[0556] The processing for separating the foreground and the backgroundexecuted by the foreground/background separator 105 is described belowwith reference to the flowchart of FIG. 74. In step S601, the framememory 621 of the separating portion 601 obtains an input image, andstores frame #n for which the foreground and the background areseparated together with the previous frame #n−1 and the subsequent frame#n+1.

[0557] In step S602, the separation processing block 622 of theseparating portion 601 obtains area information supplied from themixture-ratio calculator 104. In step S603, the separation processingblock 622 of the separating portion 601 obtains the mixture ratio αsupplied from the mixture-ratio calculator 104.

[0558] In step S604, the uncovered area processor 631 extracts thebackground components from the pixel values of the pixels belonging tothe uncovered background area supplied from the frame memory 621 basedon the area information and the mixture ratio α.

[0559] In step S605, the uncovered area processor 631 extracts theforeground components from the pixel values of the pixels belonging tothe uncovered background area supplied from the frame memory 621 basedon the area information and the mixture ratio α.

[0560] In step S606, the covered area processor 632 extracts thebackground components from the pixel values of the pixels belonging tothe covered background area supplied from the frame memory 621 based onthe area information and the mixture ratio α.

[0561] In step S607, the covered area processor 632 extracts theforeground components from the pixel values of the pixels belonging tothe covered background area supplied from the frame memory 621 based onthe area information and the mixture ratio α.

[0562] In step S608, the synthesizer 633 combines the foregroundcomponents of the pixels belonging to the uncovered background areaextracted in the processing of step S605 with the foreground componentsof the pixels belonging to the covered background area extracted in theprocessing of step S607. The synthesized foreground components aresupplied to the synthesizer 603. The synthesizer 603 further combinesthe pixels belonging to the foreground area supplied via the switch 602with the foreground components supplied from the separating portion 601so as to generate a foreground component image.

[0563] In step S609, the synthesizer 634 combines the backgroundcomponents of the pixels belonging to the uncovered background areaextracted in the processing of step S604 with the background componentsof the pixels belonging to the covered background area extracted in theprocessing of step S606. The synthesized background components aresupplied to the synthesizer 605. The synthesizer 605 further combinesthe pixels belonging to the background area supplied via the switch 604with the background components supplied from the separating portion 601so as to generate a background component image.

[0564] In step S610, the synthesizer 603 outputs the foregroundcomponent image. In step S611, the synthesizer 605 outputs thebackground component image. The processing is then completed.

[0565] As discussed above, the foreground/background separator 105 isable to separate the foreground components and the background componentsfrom the input image based on the area information and the mixture ratioα, and outputs the foreground component image consisting of only theforeground components and the background component image consisting ofonly the background components.

[0566]FIG. 75 is a block diagram illustrating an example of theconfiguration of the motion-blur adjusting unit 106. The motion vectorand the positional information thereof supplied from the motion detector102 are supplied to a unit-of-processing determining portion 901 and anadjusting portion 905. The area information supplied from the areaspecifying unit 103 is supplied to the unit-of-processing determiningportion 901. The foreground component image supplied from theforeground/background separator 105 is supplied to a calculator 904.

[0567] The unit-of-processing determining portion 901 supplies, togetherwith the motion vector, the unit of processing that is generated basedon the motion vector and the positional information thereof and the areainformation to a model-forming portion 902.

[0568] As indicated by A in FIG. 76, for example, the unit of processinggenerated by the unit-of-processing determining portion 901 indicatesconsecutive pixels disposed in the moving direction starting from thepixel corresponding to the covered background area of the foregroundcomponent image until the pixel corresponding to the uncoveredbackground area, or indicates consecutive pixels disposed in the movingdirection starting from the pixel corresponding to the uncoveredbackground area until the pixel corresponding to the covered backgroundarea. The unit of processing is formed of two pieces of data whichindicate, for example, the upper left point (which is the position ofthe leftmost or the topmost pixel in the image designated by the unit ofprocessing) and the lower right point.

[0569] The model-forming portion 902 forms a model based on the motionvector and the input unit of processing. More specifically, for example,the model-forming portion 902 may store in advance a plurality of modelsin accordance with the number of pixels contained in the unit ofprocessing, the number of virtual divided portions of the pixel value inthe time direction, and the number of foreground components for eachpixel. The model-forming portion 902 then may select the model in whichthe correlation between the pixel values and the foreground componentsis designated, such as that in FIG. 77, based on the unit of processingand the number of virtual divided portions of the pixel value in thetime direction.

[0570] It is now assumed, for example, that the number of pixelscorresponding to the unit of processing is 12, and that the amount ofmovement v within the shutter time is 5. Then, the model-forming portion902 sets the number of virtual divided portions to 5, and selects amodel formed of eight types of foreground components so that theleftmost pixel contains one foreground component, the second pixel fromthe left contains two foreground components, the third pixel from theleft contains three foreground components, the fourth pixel from theleft contains four pixel components, the fifth pixel from the leftcontains five foreground components, the sixth pixel from the leftcontains five foreground components, the seventh pixel from the leftcontains five foreground components, the eighth pixel from the leftcontains five foreground components, the ninth pixel from the leftcontains four foreground components, the tenth pixel from the leftcontains three foreground components, the eleventh pixel from the leftcontains two foreground components, and the twelfth pixel from the leftcontains one foreground component.

[0571] Instead of selecting a model from the prestored models, themodel-forming portion 902 may generate a model based on the motionvector and the unit of processing when the motion vector and the unit ofprocessing are supplied.

[0572] An equation generator 903 generates an equation based on themodel supplied from the model-forming portion 902.

[0573] A description is given below, with reference to the models of theforeground component images shown in FIGS. 77 through 79, of equationsgenerated by the equation generator 903 when the number of foregroundcomponents is 8, the number of pixels corresponding to the unit ofprocessing is 12, and the amount of movement v is 5.

[0574] When the foreground components contained in the foregroundcomponent image corresponding to the shutter time/v are F01/v throughF08/v, the relationships between F01/v through F08/v and the pixelvalues C01 through C12 can be expressed by equations (78) through (89).

C01=F01/v  (78)

C02=F02/v+F01/v  (79)

C03=F03/v+F02/v+F01v  (80)

C04=F04/v+F03/v+F02/v+F01v  (81)

C05=F05/v+F04/v+F03/v+F02/v+F01v  (82)

C06=F06/v+F05/v+F04/v+F03/v+F02/v  (83)

C07=F07/v+F06/v+F05/v+F04/v+F03/v  (84)

C08=F08/v+F07/v+F06/v+F05/v+F04/v  (85)

C09=F08/v+F07/v+F06/v+F05/v  (86)

C10=F08/v+F07/v+F06/v  (87)

C11=F08/v+F07/v  (88)

C12=F08/v  (89)

[0575] By considering the pixel values C12 and C11, the pixel value C12contains only the foreground component F08/v, as expressed by equation(90), and the pixel value C11 consists of the product sum of theforeground component F08/v and the foreground component F07/v.Accordingly, the foreground component F07/v can be found by equation(91).

F08/v=C12  (90)

F07/v=C11−C12  (91)

[0576] Similarly, by considering the foreground components contained inthe pixel values C10 through C01, the foreground components F06/vthrough F01/v can be found by equations (92) through (97), respectively.

F06/v=C10−C11  (92)

F05/v=C09−C10  (93)

F04/v=C08−C09  (94)

F03/v=C07−C08+C12  (95)

F02/v=C06−C07+C11−C12  (96)

F01/v=C05−C06+C10−C11  (97)

[0577] The equation generator 903 generates the equations forcalculating the foreground components by the difference between thepixel values, as indicated by the examples of equations (90) through(97). The equation generator 903 supplies the generated equations to thecalculator 904.

[0578] The calculator 904 sets the pixel values of the foregroundcomponent image in the equations supplied from the equation generator903 so as to obtain the foreground components based on the equations inwhich the pixel values are set. For example, when equations (90) through(97) are supplied from the equation generator 903, the calculator 904sets the pixel values C05 through C12 in equations (90) through (97).

[0579] The calculator 904 calculates the foreground components based onthe equations in which the pixel values are set. For example, thecalculator 904 calculates the foreground components F01/v through F08/v,as shown in FIG. 78, based on the calculations of equations (90) through(97) in which the pixel values C05 through C12 are set. The calculator904 supplies the foreground components F01/v through F08/v to theadjusting portion 905.

[0580] The adjusting portion 905 multiplies the foreground componentssupplied from the calculator 904 by the amount of movement v containedin the motion vector supplied from the unit-of-processing determiningportion 901 so as to obtain the foreground pixel values from whichmotion blur is eliminated. For example, when the foreground componentsF01/v through F08/v are supplied from the calculator 904, the adjustingportion 905 multiples each of the foreground components F01/v throughF08/v by the amount of movement v, i.e., 5, so as to obtain theforeground pixel values F01 through F08 from which motion blur iseliminated, as shown in FIG. 79.

[0581] The adjusting portion 905 supplies the foreground component imageconsisting of the foreground pixel values without motion blur calculatedas described above to a motion-blur adder 906 and a selector 907.

[0582] The motion-blur adder 906 is able to adjust the amount of motionblur by using the amount v′ by which motion blur is adjusted, which isdifferent from the amount of movement v, for example, the amount v′ bywhich motion blur is adjusted, which is one half the value of the amountof movement v, or the amount v′ by which motion blur is adjusted, whichis irrelevant to the amount of movement v. For example, as shown in FIG.80, the motion-blur adder 906 divides the foreground pixel value Fiwithout motion blur by the amount v′ by which motion blur is adjusted soas to obtain the foreground component Fi/v′. The motion-blur adder 906then calculates the sum of the foreground components Fi/v′, therebygenerating the pixel value in which the amount of motion blur isadjusted. For example, when the amount v′ by which motion blur isadjusted is 3, the pixel value C02 is set to (F01)/v′, the pixel valueC3 is set to (F01+F02)/v′, the pixel value C04 is set to(F01+F02+F03)/v′, and the pixel value C05 is set to (F02+F03+F04)/v′.

[0583] The motion-blur adder 906 supplies the foreground component imagein which the amount of motion blur is adjusted to the selector 907.

[0584] The selector 907 selects either the foreground component imagewithout motion blur supplied from the adjusting portion 905 or theforeground component image in which the amount of motion blur isadjusted supplied from the motion-blur adder 906 based on a selectionsignal reflecting a user's selection, and outputs the selectedforeground component image.

[0585] As discussed above, the motion-blur adjusting unit 106 is able toadjust the amount of motion blur based on the selection signal and theamount v′ by which motion blur is adjusted.

[0586] The processing for adjusting the amount of motion blur of theforeground executed by the motion-blur adjusting unit 106 is describedbelow with reference to the flowchart of FIG. 81.

[0587] In step S901, the unit-of-processing determining portion 901 ofthe motion-blur adjusting unit 106 generates the unit of processingbased on the motion vector and the area information, and supplies thegenerated unit of processing to the model-forming portion 902 and theadjusting portion 905.

[0588] In step S902, the model-forming portion 902 of the motion-bluradjusting unit 106 selects or generates the model according to theamount of movement v and the unit of processing. In step S903, theequation generator 903 generates, based on the selected or generatedmodel, the equations for calculating the foreground components by thedifference between the pixel values of the foreground component image.

[0589] In step S904, the calculator 904 sets the pixel values of theforeground component image in the generated equations, and extracts theforeground components by using the difference between the pixel valuesbased on the equations in which the pixel values are set. In step S905,the calculator 904 determines whether all the foreground componentscorresponding to the unit of processing have been extracted. If it isdetermined that all the foreground components corresponding to the unitof processing have not been extracted, the process returns to step S904,and the processing for extracting the foreground components is repeated.

[0590] If it is determined in step S905 that all the foregroundcomponents corresponding to the unit of processing have been extracted,the process proceeds to step S906. In step S906, the adjusting portion905 adjusts each of the foreground components F01/v through F08/vsupplied from the calculator 904 based on the amount of movement v so asto obtain the foreground pixel values F01/v through F08/v from whichmotion blur is eliminated.

[0591] In step S907, the motion-blur adder 906 calculates the foregroundpixel values in which the amount of motion blur is adjusted, and theselector 907 selects the image without motion blur or the image in whichthe amount of motion blur is adjusted, and outputs the selected image.The processing is then completed.

[0592] As described above, the motion-blur adjusting unit 106 configuredas shown in FIG. 75 is able to adjust motion blur of the foregroundimage containing motion blur.

[0593] A known technique for partially eliminating motion blur, such asa Wiener filter, is effective when being used in the ideal state, but isnot sufficient for an actual image quantized and containing noise. Incontrast, it is proved that the motion-blur adjusting unit 106configured as shown in FIG. 75 is sufficiently effective for an actualimage quantized and containing noise. It is thus possible to eliminatemotion blur with high precision.

[0594]FIG. 82 is a block diagram illustrating another configuration ofthe function of the signal processor 12.

[0595] The elements similar to those shown in FIG. 4 are designated withlike reference numerals, and an explanation thereof is thus omitted.

[0596] The area specifying unit 103 supplies area information to themixture-ratio calculator 104 and a synthesizer 1001.

[0597] The mixture-ratio calculator 104 supplies the mixture ratio α tothe foreground/background separator 105 and the synthesizer 1001.

[0598] The foreground/background separator 105 supplies the foregroundcomponent image to the synthesizer 1001.

[0599] The synthesizer 1001 combines a certain background image with theforeground component image supplied from the foreground/backgroundseparator 105 based on the mixture ratio α supplied from themixture-ratio calculator 104 and the area information supplied from thearea specifying unit 103, and outputs the synthesized image in which thecertain background image and the foreground component image arecombined.

[0600]FIG. 83 illustrates the configuration of the synthesizer 1001. Abackground component generator 1021 generates a background componentimage based on the mixture ratio α and a certain background image, andsupplies the background component image to a mixed-area-imagesynthesizing portion 1022.

[0601] The mixed-area-image synthesizing portion 1022 combines thebackground component image supplied from the background componentgenerator 1021 with the foreground component image so as to generate amixed-area synthesized image, and supplies the generated mixture-areasynthesized image to an image synthesizing portion 1023.

[0602] The image synthesizer 1023 combines the foreground componentimage, the mixed-area synthesized image supplied from themixed-area-image synthesizing portion 1022, and the certain backgroundimage based on the area information so as to generate a synthesizedimage, and outputs it.

[0603] As discussed above, the synthesizer 1001 is able to combine theforeground component image with a certain background image.

[0604] The image obtained by combining a foreground component image witha certain background image based on the mixture ratio α, which is thefeature quantity, appears more natural compared to an image obtained bysimply combining pixels.

[0605]FIG. 84 is a block diagram illustrating still anotherconfiguration of the function of the signal processor 12 for adjustingthe amount of motion blur. The signal processor 12 shown in FIG. 4sequentially performs the area-specifying operation and the calculationfor the mixture ratio α. In contrast, the signal processor 12 shown inFIG. 84 simultaneously performs the area-specifying operation and thecalculation for the mixture ratio α.

[0606] The functional elements similar to those in the block diagram ofFIG. 4 are designated with like reference numerals, and an explanationthereof is thus omitted.

[0607] An input image is supplied to a mixture-ratio calculator 1101, aforeground/background separator 1102, the area specifying unit 103, andthe object extracting unit 101.

[0608] The mixture-ratio calculator 1101 calculates, based on the inputimage, the estimated mixture ratio when it is assumed that each pixelcontained in the input image belongs to the covered background area, andthe estimated mixture ratio when it is assumed that each pixel containedin the input image belongs to the uncovered background area, andsupplies the estimated mixture ratios calculated as described above tothe foreground/background separator 1102.

[0609]FIG. 85 is a block diagram illustrating an example of theconfiguration of the mixture-ratio calculator 1101.

[0610] An estimated-mixture-ratio processor 401 shown in FIG. 85 is thesame as the estimated-mixture-ratio processor 401 shown in FIG. 49. Anestimated-mixture-ratio processor 402 shown in FIG. 85 is the same asthe estimated-mixture-ratio processor 402 shown in FIG. 49.

[0611] The estimated-mixture-ratio processor 401 calculates theestimated mixture ratio for each pixel by the computation correspondingto a model of the covered background area based on the input image, andoutputs the calculated estimated mixture ratio.

[0612] The estimated-mixture-ratio processor 402 calculates theestimated mixture ratio for each pixel by the computation correspondingto a model of the uncovered background area based on the input image,and outputs the calculated estimated mixture ratio.

[0613] The foreground/background separator 1102 generates the foregroundcomponent image from the input image based on the estimated mixtureratio calculated when it is assumed that the pixel belongs to thecovered background area supplied from the mixture-ratio calculator 1101,the estimated mixture ratio calculated when it is assumed that the pixelbelongs to the uncovered background area supplied from the mixture-ratiocalculator 1101, and the area information supplied from the areaspecifying unit 103, and supplies the generated foreground componentimage to the motion-blur adjusting unit 106 and the selector 107.

[0614]FIG. 86 is a block diagram illustrating an example of theconfiguration of the foreground/background separator 1102.

[0615] The elements similar to those of the foreground/backgroundseparator 105 shown in FIG. 67 are indicated by like reference numerals,and an explanation thereof is thus omitted.

[0616] A selector 1121 selects, based on the area information suppliedfrom the area specifying unit 103, either the estimated mixture ratiocalculated when it is assumed that the pixel belongs to the coveredbackground area supplied from the mixture-ratio calculator 1101 or theestimated mixture ratio calculated when it is assumed that the pixelbelongs to the uncovered background area supplied from the mixture-ratiocalculator 1101, and supplies the selected estimated mixture ratio tothe separating portion 601 as the mixture ratio α.

[0617] The separating portion 601 extracts the foreground components andthe background components from the pixel values of the pixels belongingto the mixed area based on the mixture ratio α supplied from theselector 1121 and the area information, and supplies the extractedforeground components to the synthesizer 603 and also supplies thebackground components to the synthesizer 605.

[0618] The separating portion 601 can be configured similarly to thecounterpart shown in FIG. 72.

[0619] The synthesizer 603 synthesizes the foreground component imageand outputs it. The synthesizer 605 synthesizes the background componentimage and outputs it.

[0620] The motion-blur adjusting unit 106 shown in FIG. 84 can beconfigured similarly to the counterpart shown in FIG. 4. The motion-bluradjusting unit 106 adjusts the amount of motion blur contained in theforeground component image supplied from the foreground/backgroundseparator 1102 based on the area information and the motion vector, andoutputs the foreground component image in which the amount of motionblur is adjusted.

[0621] The selector 107 shown in FIG. 84 selects the foregroundcomponent image supplied from the foreground/background separator 1102or the foreground component image in which the amount of motion blur isadjusted supplied from the motion-blur adjusting unit 106 based on, forexample, a selection signal reflecting a user's selection, and outputsthe selected foreground component image.

[0622] As discussed above, the signal processor 12 shown in FIG. 84 isable to adjust the amount of motion blur contained in an imagecorresponding to a foreground object of the input image, and outputs theresulting foreground object image. As in the first embodiment, thesignal processor 12 shown in FIG. 84 is able to calculate the mixtureratio α, which is embedded information, and outputs the calculatedmixture ratio α.

[0623] The embodiment has been discussed above by setting the mixtureratio α to the ratio of the background components contained in the pixelvalues. However, the mixture ratio α may be set to the ratio of theforeground components contained in the pixel values.

[0624] The embodiment has been discussed above by setting the movingdirection of the foreground object to the direction from the left to theright. However, the moving direction is not restricted to theabove-described direction.

[0625] In the above description, a real-space image having athree-dimensional space and time axis information is projected onto atime space having a two-dimensional space and time axis information byusing a video camera. However, the present invention is not restrictedto this example, and can be applied to the following case. When agreater amount of first information in one-dimensional space isprojected onto a smaller amount of second information in atwo-dimensional space, distortion generated by the projection can becorrected, significant information can be extracted, or a more naturalimage can be synthesized.

[0626] The sensor 11 is not restricted to a CCD, and may be another typeof sensor, such as a solid-state imaging device, for example, a CMOS(Complementary Metal Oxide Semiconductor), a BBD (Bucket BrigadeDevice), a CID (Charge Injection Device), or a CPD (Charge PrimingDevice). Also, the sensor does not have to be a sensor in whichdetection devices are arranged in a matrix, and may be a sensor in whichdetection devices are arranged in one line.

[0627] A recording medium in which a program for performing the signalprocessing of the present invention is recorded may be formed of apackage medium in which the program is recorded, which is distributedfor providing the program to a user separately from the computer, asshown in FIG. 3, such as the magnetic disk 51 (including a flexibledisk), the optical disc 52 (CD-ROM (Compact Disk-Read Only Memory) and aDVD (Digital Versatile Disk)), the magneto-optical disk 53 (including MD(Mini-Disk)), or the semiconductor memory 54. The recording medium mayalso be formed of the ROM 22 or a hard disk contained in the storageunit 28 in which the program is recorded, such recording medium beingprovided to the user while being prestored in the computer.

[0628] The steps forming the program recorded in a recording medium maybe executed chronologically according to the orders described in thespecification. However, they do not have to be executed in a time-seriesmanner, and they may be executed concurrently or individually.

[0629] Industrial Applicability

[0630] According to the first invention, the amount of motion blurcontained in a detection signal of a blurred image can be adjusted.

[0631] According to the second invention, the amount of motion blur canbe adjusted.

1. An image processing apparatus for processing image data which isformed of a predetermined number of pixel data obtained by an imagingdevice including a predetermined number of pixels, each pixel having atime integrating function, said image processing apparatus comprising:unit-of-processing determining means for determining a unit ofprocessing, based on the image data and area information indicating aforeground area consisting of foreground object components which form aforeground object in the image data, a background area consisting ofbackground object components which form a background object in the imagedata, and a mixed area in which the foreground object components and thebackground object components in the image data are mixed, the mixed areaincluding a covered background area formed at a leading end in adirection in which the foreground object is moving and an uncoveredbackground area formed at a trailing end in the direction in which theforeground object is moving, the unit of processing being formed ofpixel data located on at least a straight line that coincides with themoving direction of the foreground object and ranging from an outer endof the covered background area to an outer end of the uncoveredbackground area based on the foreground area; simultaneous-equationgenerating means for generating simultaneous equations consisting of aplurality of relational expressions by setting pixel values of pixelswithin the unit of processing determined based on the unit of processingand by setting a divided known value obtained by dividing the foregroundobject components in the mixed area by a set number of divided portions;and calculation means for calculating the foreground object componentsin which the amount of motion blur is adjusted by solving thesimultaneous equations.
 2. An image processing apparatus according toclaim 1, wherein, by utilizing a characteristic in which the foregroundobject components contained in the first pixel from an end of the mixedarea are subtracted from the foreground object components contained inthe second pixel, which is located adjacent to the first pixel, from theend of the mixed area on said straight line so that the singleforeground object component corresponding to the second pixel iscalculated, said calculation means sequentially solves the simultaneousequations from the relational expressions corresponding to the pixellocated at the end, thereby calculating the foreground object componentsin which the amount of motion blur is adjusted.
 3. An image processingapparatus according to claim 1, wherein said simultaneous-equationgenerating means generates the simultaneous equations based on thenumber of divided portions in accordance with the amount of movement ofthe foreground object.
 4. An image processing method for processingimage data which is formed of a predetermined number of pixel dataobtained by an imaging device including a predetermined number ofpixels, each pixel having a time integrating function, said imageprocessing method comprising: a unit-of-processing determining step ofdetermining a unit of processing, based on the image data and areainformation indicating a foreground area consisting of foreground objectcomponents which form a foreground object in the image data, abackground area consisting of background object components which form abackground object in the image data, and a mixed area in which theforeground object components and the background object components in theimage data are mixed, the mixed area including a covered background areaformed at a leading end in a direction in which the foreground object ismoving and an uncovered background area formed at a trailing end in thedirection in which the foreground object is moving, the unit ofprocessing being formed of pixel data located on at least a straightline that coincides with the moving direction of the foreground objectand ranging from an outer end of the covered background area to an outerend of the uncovered background area based on the foreground area; asimultaneous-equation generating step of generating simultaneousequations consisting of a plurality of relational expressions by settingpixel values of pixels within the unit of processing determined based onthe unit of processing and by setting a divided known value obtained bydividing the foreground object components in the mixed area by a setnumber of divided portions; and a calculation step of calculating theforeground object components in which the amount of motion blur isadjusted by solving the simultaneous equations.
 5. An image processingmethod according to claim 4, wherein, in said calculation step, byutilizing a characteristic in which the foreground object componentscontained in the first pixel from an end of the mixed area aresubtracted from the foreground object components contained in the secondpixel, which is located adjacent to the first pixel, from the end of themixed area on said straight line so that the single foreground objectcomponent corresponding to the second pixel is calculated, thesimultaneous equations are solved from the relational expressionscorresponding to the pixel located at the end, thereby calculating theforeground object components in which the amount of motion blur isadjusted.
 6. An image processing method according to claim 4, wherein,in said simultaneous-equation generating step, the simultaneousequations are generated based on the number of divided portions inaccordance with the amount of movement of the foreground object.
 7. Arecording medium in which a computer-readable image-processing programfor processing image data which is formed of a predetermined number ofpixel data obtained by an imaging device including a predeterminednumber of pixels, each pixel having a time integrating function, isrecorded, said computer-readable image-processing program comprising: aunit-of-processing determining step of determining a unit of processing,based on the image data and area information indicating a foregroundarea consisting of foreground object components which form a foregroundobject in the image data, a background area consisting of backgroundobject components which form a background object in the image data, anda mixed area in which the foreground object components and thebackground object components in the image data are mixed, the mixed areaincluding a covered background area formed at a leading end in adirection in which the foreground object is moving and an uncoveredbackground area formed at a trailing end in the direction in which theforeground object is moving, the unit of processing being formed ofpixel data located on at least a straight line that coincides with themoving direction of the foreground object and ranging from an outer endof the covered background area to an outer end of the uncoveredbackground area based on the foreground area; a simultaneous-equationgenerating step of generating simultaneous equations consisting of aplurality of relational expressions by setting pixel values of pixelswithin the unit of processing determined based on the unit of processingand by setting a divided known value obtained by dividing the foregroundobject components in the mixed area by a set number of divided portions;and a calculation step of calculating the foreground object componentsin which the amount of motion blur is adjusted by solving thesimultaneous equations.
 8. A recording medium according to claim 7,wherein, in said calculation step, by utilizing a characteristic inwhich the foreground object components contained in the first pixel froman end of the mixed area are subtracted from the foreground objectcomponents contained in the second pixel, which is located adjacent tothe first pixel, from the end of the mixed area on the straight line sothat the single foreground object component corresponding to the secondpixel is calculated, the simultaneous equations are solved from therelational expressions corresponding to the pixel located at the end,thereby calculating the foreground object components in which the amountof motion blur is adjusted.
 9. A recording medium according to claim 7,wherein, in said simultaneous-equation generating step, the simultaneousequations are generated based on the number of divided portions inaccordance with the amount of movement of the foreground object.
 10. Animaging apparatus comprising: imaging means for outputting a subjectimage as image data which is formed of a predetermined number of pixeldata, the subject image being captured by an imaging device including apredetermined number of pixels, each pixel having a time integratingfunction; unit-of-processing determining means for determining a unit ofprocessing, based on the image data and area information indicating aforeground area consisting of foreground object components which form aforeground object in the image data, a background area consisting ofbackground object components which form a background object in the imagedata, and a mixed area in which the foreground object components and thebackground object components in the image data are mixed, the mixed areaincluding a covered background area formed at a leading end in adirection in which the foreground object is moving and an uncoveredbackground area formed at a trailing end in the direction in which theforeground object is moving, the unit of processing being formed ofpixel data located on at least a straight line that coincides with themoving direction of the foreground object and ranging from an outer endof the covered background area to an outer end of the uncoveredbackground area based on the foreground area; simultaneous-equationgenerating means for generating simultaneous equations consisting of aplurality of relational expressions by setting pixel values of pixelswithin the unit of processing determined based on the unit of processingand by setting a divided known value obtained by dividing the foregroundobject components in the mixed area by a set number of divided portions;and calculation means for calculating the foreground object componentsin which the amount of motion blur is adjusted by solving thesimultaneous equations.
 11. An imaging apparatus according to claim 10,wherein, by utilizing a characteristic in which the foreground objectcomponents contained in the first pixel from an end of the mixed areaare subtracted from the foreground object components contained in thesecond pixel, which is located adjacent to the first pixel, from the endof the mixed area on the straight line so that the single foregroundobject component corresponding to the second pixel is calculated, saidcalculation means sequentially solves the simultaneous equations fromthe relational expressions corresponding to the pixel located at theend, thereby calculating the foreground object components in which theamount of motion blur is adjusted.
 12. An imaging apparatus according toclaim 10, wherein said simultaneous-equation generating means generatesthe simultaneous equations based on the number of divided portions inaccordance with the amount of movement of the foreground object.